FIT Profiler Report

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Action                                                                                                                                                                                                     	|  Mean duration (s)	|  Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Total                                                                                                                                                                                                      	|  -              	|  881265         	|  2323.7         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  run_training_epoch                                                                                                                                                                                         	|  115.32         	|  20             	|  2306.4         	|  99.254         	|
|  run_training_batch                                                                                                                                                                                         	|  0.11014        	|  20000          	|  2202.8         	|  94.796         	|
|  [LightningModule]NeRFSystem.optimizer_step                                                                                                                                                                 	|  0.10984        	|  20000          	|  2196.8         	|  94.538         	|
|  [Strategy]SingleDeviceStrategy.training_step                                                                                                                                                               	|  0.028536       	|  20000          	|  570.73         	|  24.561         	|
|  [Strategy]SingleDeviceStrategy.backward                                                                                                                                                                    	|  0.0044613      	|  20000          	|  89.226         	|  3.8398         	|
|  [Strategy]SingleDeviceStrategy.validation_step                                                                                                                                                             	|  0.90608        	|  50             	|  45.304         	|  1.9496         	|
|  [TrainingEpochLoop].train_dataloader_next                                                                                                                                                                  	|  0.0018498      	|  20000          	|  36.997         	|  1.5921         	|
|  [Callback]TQDMProgressBar.on_train_batch_end                                                                                                                                                               	|  0.00043018     	|  20000          	|  8.6036         	|  0.37025        	|
|  [LightningModule]NeRFSystem.setup                                                                                                                                                                          	|  6.1149         	|  1              	|  6.1149         	|  0.26316        	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_end                 	|  0.23968        	|  20             	|  4.7936         	|  0.20629        	|
|  [Strategy]SingleDeviceStrategy.batch_to_device                                                                                                                                                             	|  0.00010999     	|  20050          	|  2.2053         	|  0.094907       	|
|  [LightningModule]NeRFSystem.transfer_batch_to_device                                                                                                                                                       	|  6.843e-05      	|  20050          	|  1.372          	|  0.059045       	|
|  [LightningModule]NeRFSystem.optimizer_zero_grad                                                                                                                                                            	|  2.1373e-05     	|  20000          	|  0.42746        	|  0.018396       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_end                 	|  1.398e-05      	|  20000          	|  0.27961        	|  0.012033       	|
|  [LightningModule]NeRFSystem.configure_gradient_clipping                                                                                                                                                    	|  1.3503e-05     	|  20000          	|  0.27007        	|  0.011622       	|
|  [EvaluationEpochLoop].val_dataloader_idx_0_next                                                                                                                                                            	|  0.0047057      	|  50             	|  0.23529        	|  0.010125       	|
|  [LightningModule]NeRFSystem.on_validation_start                                                                                                                                                            	|  0.077082       	|  2              	|  0.15416        	|  0.0066345      	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_end                                                                                                                                                 	|  4.2951e-06     	|  20000          	|  0.085903       	|  0.0036968      	|
|  [Callback]TQDMProgressBar.on_after_backward                                                                                                                                                                	|  3.4544e-06     	|  20000          	|  0.069089       	|  0.0029732      	|
|  [Callback]TQDMProgressBar.on_batch_start                                                                                                                                                                   	|  3.1626e-06     	|  20000          	|  0.063253       	|  0.0027221      	|
|  [Callback]TQDMProgressBar.on_before_optimizer_step                                                                                                                                                         	|  2.9987e-06     	|  20000          	|  0.059973       	|  0.0025809      	|
|  [Callback]TQDMProgressBar.on_before_zero_grad                                                                                                                                                              	|  2.8532e-06     	|  20000          	|  0.057065       	|  0.0024558      	|
|  [Callback]TQDMProgressBar.on_before_backward                                                                                                                                                               	|  2.7172e-06     	|  20000          	|  0.054345       	|  0.0023387      	|
|  [Callback]TQDMProgressBar.on_train_batch_start                                                                                                                                                             	|  2.5504e-06     	|  20000          	|  0.051009       	|  0.0021951      	|
|  [Callback]GradientAccumulationScheduler.on_batch_start                                                                                                                                                     	|  2.5139e-06     	|  20000          	|  0.050278       	|  0.0021637      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_start                     	|  2.3686e-06     	|  20000          	|  0.047372       	|  0.0020387      	|
|  [Callback]TQDMProgressBar.on_batch_end                                                                                                                                                                     	|  2.2885e-06     	|  20000          	|  0.045769       	|  0.0019697      	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_start                                                                                                                                               	|  2.2558e-06     	|  20000          	|  0.045116       	|  0.0019415      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_start               	|  2.2214e-06     	|  20000          	|  0.044427       	|  0.0019119      	|
|  [Callback]GradientAccumulationScheduler.on_before_optimizer_step                                                                                                                                           	|  2.1875e-06     	|  20000          	|  0.04375        	|  0.0018828      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_after_backward                  	|  2.1368e-06     	|  20000          	|  0.042736       	|  0.0018391      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_end                       	|  2.0562e-06     	|  20000          	|  0.041125       	|  0.0017698      	|
|  [Callback]GradientAccumulationScheduler.on_before_zero_grad                                                                                                                                                	|  2.0331e-06     	|  20000          	|  0.040661       	|  0.0017498      	|
|  [Callback]GradientAccumulationScheduler.on_after_backward                                                                                                                                                  	|  2.0108e-06     	|  20000          	|  0.040216       	|  0.0017307      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_optimizer_step           	|  2.0047e-06     	|  20000          	|  0.040093       	|  0.0017254      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_zero_grad                	|  1.9827e-06     	|  20000          	|  0.039654       	|  0.0017065      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_backward                 	|  1.9776e-06     	|  20000          	|  0.039552       	|  0.0017021      	|
|  [Callback]GradientAccumulationScheduler.on_batch_end                                                                                                                                                       	|  1.9649e-06     	|  20000          	|  0.039299       	|  0.0016912      	|
|  [Callback]GradientAccumulationScheduler.on_before_backward                                                                                                                                                 	|  1.9364e-06     	|  20000          	|  0.038728       	|  0.0016666      	|
|  [Callback]TQDMProgressBar.on_validation_batch_end                                                                                                                                                          	|  0.0004904      	|  50             	|  0.02452        	|  0.0010552      	|
|  [LightningModule]NeRFSystem.on_before_batch_transfer                                                                                                                                                       	|  1.1774e-06     	|  20050          	|  0.023606       	|  0.0010159      	|
|  [LightningModule]NeRFSystem.training_step_end                                                                                                                                                              	|  1.0846e-06     	|  20000          	|  0.021691       	|  0.00093347     	|
|  [LightningModule]NeRFSystem.on_after_batch_transfer                                                                                                                                                        	|  1.0139e-06     	|  20050          	|  0.020329       	|  0.00087487     	|
|  [Strategy]SingleDeviceStrategy.training_step_end                                                                                                                                                           	|  9.9026e-07     	|  20000          	|  0.019805       	|  0.00085231     	|
|  [LightningModule]NeRFSystem.on_train_batch_end                                                                                                                                                             	|  9.64e-07       	|  20000          	|  0.01928        	|  0.00082971     	|
|  [LightningModule]NeRFSystem.on_train_batch_start                                                                                                                                                           	|  8.9792e-07     	|  20000          	|  0.017958       	|  0.00077284     	|
|  [LightningModule]NeRFSystem.on_before_zero_grad                                                                                                                                                            	|  8.2489e-07     	|  20000          	|  0.016498       	|  0.00070998     	|
|  [Strategy]SingleDeviceStrategy.on_train_batch_start                                                                                                                                                        	|  8.1685e-07     	|  20000          	|  0.016337       	|  0.00070306     	|
|  [LightningModule]NeRFSystem.on_before_backward                                                                                                                                                             	|  7.7906e-07     	|  20000          	|  0.015581       	|  0.00067054     	|
|  [LightningModule]NeRFSystem.on_before_optimizer_step                                                                                                                                                       	|  7.1909e-07     	|  20000          	|  0.014382       	|  0.00061892     	|
|  [LightningModule]NeRFSystem.on_after_backward                                                                                                                                                              	|  7.0205e-07     	|  20000          	|  0.014041       	|  0.00060426     	|
|  [LightningModule]NeRFSystem.validation_epoch_end                                                                                                                                                           	|  0.0048791      	|  2              	|  0.0097581      	|  0.00041994     	|
|  [LightningModule]NeRFSystem.configure_optimizers                                                                                                                                                           	|  0.0078962      	|  1              	|  0.0078962      	|  0.00033981     	|
|  [Callback]TQDMProgressBar.on_train_epoch_end                                                                                                                                                               	|  0.0002227      	|  20             	|  0.0044541      	|  0.00019168     	|
|  [Callback]TQDMProgressBar.on_train_epoch_start                                                                                                                                                             	|  0.00021925     	|  20             	|  0.0043849      	|  0.0001887      	|
|  [Callback]TQDMProgressBar.on_validation_batch_start                                                                                                                                                        	|  4.5575e-05     	|  50             	|  0.0022787      	|  9.8065e-05     	|
|  [LightningModule]NeRFSystem.lr_scheduler_step                                                                                                                                                              	|  9.9248e-05     	|  20             	|  0.001985       	|  8.5422e-05     	|
|  [Callback]TQDMProgressBar.on_validation_end                                                                                                                                                                	|  0.00049216     	|  2              	|  0.00098431     	|  4.236e-05      	|
|  [LightningModule]NeRFSystem.on_validation_model_train                                                                                                                                                      	|  0.0004899      	|  2              	|  0.00097979     	|  4.2165e-05     	|
|  [LightningModule]NeRFSystem.on_validation_model_eval                                                                                                                                                       	|  0.00040246     	|  2              	|  0.00080493     	|  3.464e-05      	|
|  [Callback]TQDMProgressBar.on_train_end                                                                                                                                                                     	|  0.00069137     	|  1              	|  0.00069137     	|  2.9753e-05     	|
|  [LightningModule]NeRFSystem.train_dataloader                                                                                                                                                               	|  0.00058223     	|  1              	|  0.00058223     	|  2.5056e-05     	|
|  [Callback]TQDMProgressBar.on_validation_start                                                                                                                                                              	|  0.00019482     	|  2              	|  0.00038965     	|  1.6768e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_end                                                                                                                                            	|  7.5889e-06     	|  50             	|  0.00037945     	|  1.6329e-05     	|
|  [Callback]TQDMProgressBar.on_train_start                                                                                                                                                                   	|  0.00026432     	|  1              	|  0.00026432     	|  1.1375e-05     	|
|  [LightningModule]NeRFSystem.val_dataloader                                                                                                                                                                 	|  0.00025221     	|  1              	|  0.00025221     	|  1.0854e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_start                                                                                                                                          	|  4.2816e-06     	|  50             	|  0.00021408     	|  9.2128e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_end            	|  3.4869e-06     	|  50             	|  0.00017435     	|  7.5029e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_start          	|  3.0985e-06     	|  50             	|  0.00015493     	|  6.6672e-06     	|
|  [LightningModule]NeRFSystem.validation_step_end                                                                                                                                                            	|  3.0969e-06     	|  50             	|  0.00015484     	|  6.6637e-06     	|
|  [LightningModule]NeRFSystem.on_validation_batch_end                                                                                                                                                        	|  2.342e-06      	|  50             	|  0.0001171      	|  5.0394e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_end                  	|  5.5969e-05     	|  2              	|  0.00011194     	|  4.8172e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_start                                                                                                                                               	|  5.501e-06      	|  20             	|  0.00011002     	|  4.7347e-06     	|
|  [LightningModule]NeRFSystem.on_validation_batch_start                                                                                                                                                      	|  1.9258e-06     	|  50             	|  9.6292e-05     	|  4.1439e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_end                                                                                                                                                 	|  4.4589e-06     	|  20             	|  8.9178e-05     	|  3.8378e-06     	|
|  [Strategy]SingleDeviceStrategy.validation_step_end                                                                                                                                                         	|  1.4578e-06     	|  50             	|  7.2888e-05     	|  3.1367e-06     	|
|  [Callback]TQDMProgressBar.on_epoch_start                                                                                                                                                                   	|  3.0669e-06     	|  22             	|  6.7472e-05     	|  2.9036e-06     	|
|  [Callback]TQDMProgressBar.on_epoch_end                                                                                                                                                                     	|  2.9488e-06     	|  22             	|  6.4873e-05     	|  2.7918e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_start               	|  2.6554e-06     	|  20             	|  5.3109e-05     	|  2.2855e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end                       	|  2.1466e-06     	|  22             	|  4.7225e-05     	|  2.0323e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_end                                                                                                                                                       	|  2.1345e-06     	|  22             	|  4.696e-05      	|  2.0209e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_start                                                                                                                                                     	|  1.991e-06      	|  22             	|  4.3803e-05     	|  1.885e-06      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start                     	|  1.9851e-06     	|  22             	|  4.3672e-05     	|  1.8794e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_end                                                                                                                                                             	|  1.4785e-06     	|  20             	|  2.9569e-05     	|  1.2725e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.setup                              	|  2.944e-05      	|  1              	|  2.944e-05      	|  1.2669e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_end                                                                                                                                                                   	|  1.2859e-06     	|  22             	|  2.8291e-05     	|  1.2175e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_start                                                                                                                                                                 	|  1.206e-06      	|  22             	|  2.6532e-05     	|  1.1418e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_start                                                                                                                                                           	|  1.167e-06      	|  20             	|  2.334e-05      	|  1.0044e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_end                                                                                                                                                  	|  9.9996e-06     	|  2              	|  1.9999e-05     	|  8.6066e-07     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_end                                                                                                                                                          	|  9.8352e-06     	|  2              	|  1.967e-05      	|  8.4651e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_end                                                                                                                                                        	|  8.6445e-06     	|  2              	|  1.7289e-05     	|  7.4403e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_end                                                                                                                                                           	|  5.6e-06        	|  2              	|  1.12e-05       	|  4.8199e-07     	|
|  [Callback]TQDMProgressBar.setup                                                                                                                                                                            	|  9.2397e-06     	|  1              	|  9.2397e-06     	|  3.9763e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_end                                                                                                                                                       	|  9.2303e-06     	|  1              	|  9.2303e-06     	|  3.9723e-07     	|
|  [Callback]TQDMProgressBar.on_fit_end                                                                                                                                                                       	|  8.5598e-06     	|  1              	|  8.5598e-06     	|  3.6837e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_start                                                                                                                                                	|  4.2301e-06     	|  2              	|  8.4601e-06     	|  3.6408e-07     	|
|  [LightningModule]NeRFSystem.configure_callbacks                                                                                                                                                            	|  7.6098e-06     	|  1              	|  7.6098e-06     	|  3.2749e-07     	|
|  [Callback]TQDMProgressBar.on_configure_sharded_model                                                                                                                                                       	|  6.7698e-06     	|  1              	|  6.7698e-06     	|  2.9134e-07     	|
|  [Callback]TQDMProgressBar.on_fit_start                                                                                                                                                                     	|  6.5397e-06     	|  1              	|  6.5397e-06     	|  2.8144e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_end                                                                                                                                            	|  3.0049e-06     	|  2              	|  6.0098e-06     	|  2.5863e-07     	|
|  [Callback]GradientAccumulationScheduler.setup                                                                                                                                                              	|  5.7099e-06     	|  1              	|  5.7099e-06     	|  2.4573e-07     	|
|  [Callback]TQDMProgressBar.teardown                                                                                                                                                                         	|  5.3896e-06     	|  1              	|  5.3896e-06     	|  2.3194e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_end            	|  2.6501e-06     	|  2              	|  5.3002e-06     	|  2.2809e-07     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_start                                                                                                                                                        	|  2.61e-06       	|  2              	|  5.2201e-06     	|  2.2464e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_end                                                                                                                                                         	|  4.95e-06       	|  1              	|  4.95e-06       	|  2.1302e-07     	|
|  [Callback]GradientAccumulationScheduler.on_configure_sharded_model                                                                                                                                         	|  4.8997e-06     	|  1              	|  4.8997e-06     	|  2.1086e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_end                       	|  4.8801e-06     	|  1              	|  4.8801e-06     	|  2.1002e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model         	|  4.6305e-06     	|  1              	|  4.6305e-06     	|  1.9927e-07     	|
|  [Callback]GradientAccumulationScheduler.teardown                                                                                                                                                           	|  4.57e-06       	|  1              	|  4.57e-06       	|  1.9667e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_start                                                                                                                                                     	|  4.53e-06       	|  1              	|  4.53e-06       	|  1.9495e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_start                     	|  4.4005e-06     	|  1              	|  4.4005e-06     	|  1.8937e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_end                         	|  4.3493e-06     	|  1              	|  4.3493e-06     	|  1.8717e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_start                                                                                                                                                        	|  4.2301e-06     	|  1              	|  4.2301e-06     	|  1.8204e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown                           	|  4.2198e-06     	|  1              	|  4.2198e-06     	|  1.816e-07      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_start                	|  2.1053e-06     	|  2              	|  4.2105e-06     	|  1.812e-07      	|
|  [LightningModule]NeRFSystem.on_fit_start                                                                                                                                                                   	|  4.0792e-06     	|  1              	|  4.0792e-06     	|  1.7555e-07     	|
|  [LightningModule]NeRFSystem.configure_sharded_model                                                                                                                                                        	|  4.0103e-06     	|  1              	|  4.0103e-06     	|  1.7258e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_start                                                                                                                                                       	|  3.8901e-06     	|  1              	|  3.8901e-06     	|  1.6741e-07     	|
|  [LightningModule]NeRFSystem.on_validation_end                                                                                                                                                              	|  1.9399e-06     	|  2              	|  3.8799e-06     	|  1.6697e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_start          	|  1.8901e-06     	|  2              	|  3.7802e-06     	|  1.6268e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_start                                                                                                                                          	|  1.8799e-06     	|  2              	|  3.7597e-06     	|  1.618e-07      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_start                       	|  3.6703e-06     	|  1              	|  3.6703e-06     	|  1.5795e-07     	|
|  [Callback]TQDMProgressBar.on_before_accelerator_backend_setup                                                                                                                                              	|  3.2708e-06     	|  1              	|  3.2708e-06     	|  1.4076e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_start                                                                                                                                                         	|  1.5302e-06     	|  2              	|  3.0603e-06     	|  1.317e-07      	|
|  [LightningModule]NeRFSystem.teardown                                                                                                                                                                       	|  2.9104e-06     	|  1              	|  2.9104e-06     	|  1.2525e-07     	|
|  [LightningModule]NeRFSystem.on_train_end                                                                                                                                                                   	|  2.78e-06       	|  1              	|  2.78e-06       	|  1.1964e-07     	|
|  [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup                                                                                                                                	|  2.6701e-06     	|  1              	|  2.6701e-06     	|  1.1491e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_accelerator_backend_setup	|  2.6207e-06     	|  1              	|  2.6207e-06     	|  1.1278e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_end                                                                                                                                                          	|  2.55e-06       	|  1              	|  2.55e-06       	|  1.0974e-07     	|
|  [LightningModule]NeRFSystem.on_fit_end                                                                                                                                                                     	|  2.3898e-06     	|  1              	|  2.3898e-06     	|  1.0284e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_start                                                                                                                                          	|  2.2706e-06     	|  1              	|  2.2706e-06     	|  9.7713e-08     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_start          	|  2.1495e-06     	|  1              	|  2.1495e-06     	|  9.2503e-08     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_start                                                                                                                                                      	|  1.0449e-06     	|  2              	|  2.0899e-06     	|  8.9938e-08     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_end            	|  1.9697e-06     	|  1              	|  1.9697e-06     	|  8.4768e-08     	|
|  [LightningModule]NeRFSystem.prepare_data                                                                                                                                                                   	|  1.9101e-06     	|  1              	|  1.9101e-06     	|  8.2203e-08     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_end                                                                                                                                            	|  1.8701e-06     	|  1              	|  1.8701e-06     	|  8.0479e-08     	|
|  [LightningModule]NeRFSystem.on_save_checkpoint                                                                                                                                                             	|  1.6503e-06     	|  1              	|  1.6503e-06     	|  7.102e-08      	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_start                                                                                                                                                      	|  1.6401e-06     	|  1              	|  1.6401e-06     	|  7.058e-08      	|
|  [Strategy]SingleDeviceStrategy.on_train_end                                                                                                                                                                	|  1.6401e-06     	|  1              	|  1.6401e-06     	|  7.058e-08      	|
|  [LightningModule]NeRFSystem.on_train_start                                                                                                                                                                 	|  1.1697e-06     	|  1              	|  1.1697e-06     	|  5.034e-08      	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_end                                                                                                                                                        	|  1.1502e-06     	|  1              	|  1.1502e-06     	|  4.9498e-08     	|
|  [Strategy]SingleDeviceStrategy.on_train_start                                                                                                                                                              	|  8.6986e-07     	|  1              	|  8.6986e-07     	|  3.7434e-08     	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
