FIT Profiler Report

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Action                                                                                                                                                                                                     	|  Mean duration (s)	|  Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  Total                                                                                                                                                                                                      	|  -              	|  881585         	|  1135.2         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|  run_training_epoch                                                                                                                                                                                         	|  53.586         	|  20             	|  1071.7         	|  94.41          	|
|  run_training_batch                                                                                                                                                                                         	|  0.041484       	|  20000          	|  829.68         	|  73.089         	|
|  [LightningModule]NeRFSystem.optimizer_step                                                                                                                                                                 	|  0.041102       	|  20000          	|  822.04         	|  72.416         	|
|  [Strategy]SingleDeviceStrategy.training_step                                                                                                                                                               	|  0.014976       	|  20000          	|  299.51         	|  26.385         	|
|  [TrainingEpochLoop].train_dataloader_next                                                                                                                                                                  	|  0.0090908      	|  20000          	|  181.82         	|  16.017         	|
|  [LightningModule]NeRFSystem.setup                                                                                                                                                                          	|  47.028         	|  1              	|  47.028         	|  4.1428         	|
|  [Strategy]SingleDeviceStrategy.backward                                                                                                                                                                    	|  0.0023263      	|  20000          	|  46.526         	|  4.0986         	|
|  [Strategy]SingleDeviceStrategy.validation_step                                                                                                                                                             	|  0.35951        	|  70             	|  25.166         	|  2.2169         	|
|  [Callback]TQDMProgressBar.on_train_batch_end                                                                                                                                                               	|  0.00069377     	|  20000          	|  13.875         	|  1.2223         	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_end                 	|  0.20767        	|  20             	|  4.1533         	|  0.36588        	|
|  [Strategy]SingleDeviceStrategy.batch_to_device                                                                                                                                                             	|  0.00020107     	|  20070          	|  4.0355         	|  0.3555         	|
|  [LightningModule]NeRFSystem.transfer_batch_to_device                                                                                                                                                       	|  0.00012714     	|  20070          	|  2.5516         	|  0.22478        	|
|  [LightningModule]NeRFSystem.optimizer_zero_grad                                                                                                                                                            	|  3.4112e-05     	|  20000          	|  0.68224        	|  0.0601         	|
|  [EvaluationEpochLoop].val_dataloader_idx_0_next                                                                                                                                                            	|  0.0091768      	|  70             	|  0.64237        	|  0.056589       	|
|  [LightningModule]NeRFSystem.configure_gradient_clipping                                                                                                                                                    	|  2.2671e-05     	|  20000          	|  0.45343        	|  0.039944       	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_end                 	|  2.2319e-05     	|  20000          	|  0.44638        	|  0.039323       	|
|  [LightningModule]NeRFSystem.on_validation_start                                                                                                                                                            	|  0.12107        	|  2              	|  0.24214        	|  0.021331       	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_end                                                                                                                                                 	|  5.0197e-06     	|  20000          	|  0.10039        	|  0.008844       	|
|  [Callback]TQDMProgressBar.on_after_backward                                                                                                                                                                	|  4.6001e-06     	|  20000          	|  0.092001       	|  0.0081046      	|
|  [Callback]TQDMProgressBar.on_batch_start                                                                                                                                                                   	|  4.1382e-06     	|  20000          	|  0.082764       	|  0.0072909      	|
|  [Callback]TQDMProgressBar.on_before_optimizer_step                                                                                                                                                         	|  3.566e-06      	|  20000          	|  0.07132        	|  0.0062828      	|
|  [Callback]TQDMProgressBar.on_before_zero_grad                                                                                                                                                              	|  3.5552e-06     	|  20000          	|  0.071103       	|  0.0062637      	|
|  [Callback]TQDMProgressBar.on_before_backward                                                                                                                                                               	|  3.3378e-06     	|  20000          	|  0.066756       	|  0.0058807      	|
|  [Callback]TQDMProgressBar.on_train_batch_start                                                                                                                                                             	|  3.2559e-06     	|  20000          	|  0.065118       	|  0.0057364      	|
|  [Callback]TQDMProgressBar.on_batch_end                                                                                                                                                                     	|  3.0534e-06     	|  20000          	|  0.061068       	|  0.0053797      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_batch_start               	|  2.9751e-06     	|  20000          	|  0.059502       	|  0.0052417      	|
|  [Callback]GradientAccumulationScheduler.on_batch_start                                                                                                                                                     	|  2.9679e-06     	|  20000          	|  0.059359       	|  0.0052291      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_start                     	|  2.9607e-06     	|  20000          	|  0.059215       	|  0.0052164      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_after_backward                  	|  2.7618e-06     	|  20000          	|  0.055236       	|  0.0048659      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_optimizer_step           	|  2.6551e-06     	|  20000          	|  0.053102       	|  0.0046779      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_backward                 	|  2.6175e-06     	|  20000          	|  0.05235        	|  0.0046116      	|
|  [Callback]GradientAccumulationScheduler.on_after_backward                                                                                                                                                  	|  2.6152e-06     	|  20000          	|  0.052303       	|  0.0046076      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_zero_grad                	|  2.572e-06      	|  20000          	|  0.05144        	|  0.0045315      	|
|  [Callback]GradientAccumulationScheduler.on_train_batch_start                                                                                                                                               	|  2.5196e-06     	|  20000          	|  0.050391       	|  0.0044391      	|
|  [Callback]GradientAccumulationScheduler.on_before_zero_grad                                                                                                                                                	|  2.4731e-06     	|  20000          	|  0.049463       	|  0.0043573      	|
|  [Callback]GradientAccumulationScheduler.on_before_optimizer_step                                                                                                                                           	|  2.3738e-06     	|  20000          	|  0.047476       	|  0.0041823      	|
|  [LightningModule]NeRFSystem.on_after_batch_transfer                                                                                                                                                        	|  2.3639e-06     	|  20070          	|  0.047444       	|  0.0041795      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_batch_end                       	|  2.2897e-06     	|  20000          	|  0.045795       	|  0.0040342      	|
|  [Callback]GradientAccumulationScheduler.on_before_backward                                                                                                                                                 	|  2.1629e-06     	|  20000          	|  0.043258       	|  0.0038107      	|
|  [LightningModule]NeRFSystem.on_before_batch_transfer                                                                                                                                                       	|  2.1484e-06     	|  20070          	|  0.043118       	|  0.0037984      	|
|  [Callback]GradientAccumulationScheduler.on_batch_end                                                                                                                                                       	|  2.1517e-06     	|  20000          	|  0.043035       	|  0.0037911      	|
|  [LightningModule]NeRFSystem.on_train_batch_end                                                                                                                                                             	|  1.9793e-06     	|  20000          	|  0.039586       	|  0.0034873      	|
|  [LightningModule]NeRFSystem.on_train_batch_start                                                                                                                                                           	|  1.615e-06      	|  20000          	|  0.0323         	|  0.0028454      	|
|  [LightningModule]NeRFSystem.training_step_end                                                                                                                                                              	|  1.5795e-06     	|  20000          	|  0.03159        	|  0.0027829      	|
|  [Callback]TQDMProgressBar.on_validation_batch_end                                                                                                                                                          	|  0.00043868     	|  70             	|  0.030708       	|  0.0027051      	|
|  [Strategy]SingleDeviceStrategy.on_train_batch_start                                                                                                                                                        	|  1.3927e-06     	|  20000          	|  0.027854       	|  0.0024537      	|
|  [LightningModule]NeRFSystem.on_before_zero_grad                                                                                                                                                            	|  1.2637e-06     	|  20000          	|  0.025274       	|  0.0022265      	|
|  [Strategy]SingleDeviceStrategy.training_step_end                                                                                                                                                           	|  1.2615e-06     	|  20000          	|  0.025229       	|  0.0022225      	|
|  [LightningModule]NeRFSystem.on_after_backward                                                                                                                                                              	|  1.2001e-06     	|  20000          	|  0.024001       	|  0.0021143      	|
|  [LightningModule]NeRFSystem.on_before_optimizer_step                                                                                                                                                       	|  1.1884e-06     	|  20000          	|  0.023768       	|  0.0020938      	|
|  [LightningModule]NeRFSystem.on_before_backward                                                                                                                                                             	|  1.1642e-06     	|  20000          	|  0.023285       	|  0.0020512      	|
|  [LightningModule]NeRFSystem.validation_epoch_end                                                                                                                                                           	|  0.0052202      	|  2              	|  0.01044        	|  0.00091973     	|
|  [Callback]TQDMProgressBar.on_train_epoch_end                                                                                                                                                               	|  0.00033101     	|  20             	|  0.0066202      	|  0.00058319     	|
|  [Callback]TQDMProgressBar.on_train_epoch_start                                                                                                                                                             	|  0.0002731      	|  20             	|  0.0054621      	|  0.00048117     	|
|  [LightningModule]NeRFSystem.configure_optimizers                                                                                                                                                           	|  0.0048266      	|  1              	|  0.0048266      	|  0.00042519     	|
|  [Callback]TQDMProgressBar.on_validation_batch_start                                                                                                                                                        	|  4.9659e-05     	|  70             	|  0.0034761      	|  0.00030622     	|
|  [LightningModule]NeRFSystem.lr_scheduler_step                                                                                                                                                              	|  0.00012469     	|  20             	|  0.0024939      	|  0.00021969     	|
|  [Callback]TQDMProgressBar.on_validation_end                                                                                                                                                                	|  0.00087568     	|  2              	|  0.0017514      	|  0.00015428     	|
|  [LightningModule]NeRFSystem.on_validation_model_eval                                                                                                                                                       	|  0.00069389     	|  2              	|  0.0013878      	|  0.00012225     	|
|  [LightningModule]NeRFSystem.on_validation_model_train                                                                                                                                                      	|  0.00068899     	|  2              	|  0.001378       	|  0.00012139     	|
|  [Callback]TQDMProgressBar.on_validation_start                                                                                                                                                              	|  0.00055075     	|  2              	|  0.0011015      	|  9.7035e-05     	|
|  [LightningModule]NeRFSystem.train_dataloader                                                                                                                                                               	|  0.00084295     	|  1              	|  0.00084295     	|  7.4258e-05     	|
|  [LightningModule]NeRFSystem.val_dataloader                                                                                                                                                                 	|  0.00041074     	|  1              	|  0.00041074     	|  3.6183e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_end                                                                                                                                            	|  5.8434e-06     	|  70             	|  0.00040904     	|  3.6034e-05     	|
|  [Callback]TQDMProgressBar.on_train_start                                                                                                                                                                   	|  0.0004039      	|  1              	|  0.0004039      	|  3.558e-05      	|
|  [Callback]TQDMProgressBar.on_train_end                                                                                                                                                                     	|  0.00027688     	|  1              	|  0.00027688     	|  2.4391e-05     	|
|  [Callback]GradientAccumulationScheduler.on_validation_batch_start                                                                                                                                          	|  3.5009e-06     	|  70             	|  0.00024506     	|  2.1588e-05     	|
|  [LightningModule]NeRFSystem.validation_step_end                                                                                                                                                            	|  3.0211e-06     	|  70             	|  0.00021148     	|  1.863e-05      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_end            	|  2.9076e-06     	|  70             	|  0.00020353     	|  1.793e-05      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_batch_start          	|  2.6697e-06     	|  70             	|  0.00018688     	|  1.6463e-05     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_start                                                                                                                                               	|  6.0728e-06     	|  20             	|  0.00012146     	|  1.0699e-05     	|
|  [LightningModule]NeRFSystem.on_validation_batch_start                                                                                                                                                      	|  1.5631e-06     	|  70             	|  0.00010942     	|  9.6391e-06     	|
|  [LightningModule]NeRFSystem.on_validation_batch_end                                                                                                                                                        	|  1.5604e-06     	|  70             	|  0.00010923     	|  9.6221e-06     	|
|  [Callback]GradientAccumulationScheduler.on_train_epoch_end                                                                                                                                                 	|  5.048e-06      	|  20             	|  0.00010096     	|  8.8938e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_end                  	|  4.8885e-05     	|  2              	|  9.7769e-05     	|  8.6128e-06     	|
|  [Strategy]SingleDeviceStrategy.validation_step_end                                                                                                                                                         	|  1.2607e-06     	|  70             	|  8.825e-05      	|  7.7742e-06     	|
|  [Callback]TQDMProgressBar.on_epoch_start                                                                                                                                                                   	|  3.6158e-06     	|  22             	|  7.9548e-05     	|  7.0076e-06     	|
|  [Callback]TQDMProgressBar.on_epoch_end                                                                                                                                                                     	|  3.314e-06      	|  22             	|  7.2909e-05     	|  6.4227e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_epoch_start               	|  3.1218e-06     	|  20             	|  6.2437e-05     	|  5.5002e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_start                                                                                                                                                     	|  2.269e-06      	|  22             	|  4.9919e-05     	|  4.3975e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end                       	|  2.2158e-06     	|  22             	|  4.8748e-05     	|  4.2944e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start                     	|  2.1959e-06     	|  22             	|  4.831e-05      	|  4.2558e-06     	|
|  [Callback]GradientAccumulationScheduler.on_epoch_end                                                                                                                                                       	|  2.1705e-06     	|  22             	|  4.7751e-05     	|  4.2065e-06     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.setup                              	|  4.454e-05      	|  1              	|  4.454e-05      	|  3.9236e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_start                                                                                                                                                                 	|  1.5829e-06     	|  22             	|  3.4823e-05     	|  3.0677e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_end                                                                                                                                                             	|  1.6754e-06     	|  20             	|  3.3509e-05     	|  2.9519e-06     	|
|  [LightningModule]NeRFSystem.on_epoch_end                                                                                                                                                                   	|  1.4264e-06     	|  22             	|  3.1381e-05     	|  2.7644e-06     	|
|  [LightningModule]NeRFSystem.on_train_epoch_start                                                                                                                                                           	|  1.5569e-06     	|  20             	|  3.1139e-05     	|  2.7431e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_end                                                                                                                                                  	|  1.3225e-05     	|  2              	|  2.645e-05      	|  2.33e-06       	|
|  [LightningModule]NeRFSystem.on_validation_epoch_end                                                                                                                                                        	|  7.0846e-06     	|  2              	|  1.4169e-05     	|  1.2482e-06     	|
|  [LightningModule]NeRFSystem.configure_callbacks                                                                                                                                                            	|  1.3661e-05     	|  1              	|  1.3661e-05     	|  1.2034e-06     	|
|  [Callback]GradientAccumulationScheduler.on_validation_start                                                                                                                                                	|  5.8548e-06     	|  2              	|  1.171e-05      	|  1.0315e-06     	|
|  [Callback]TQDMProgressBar.setup                                                                                                                                                                            	|  1.0311e-05     	|  1              	|  1.0311e-05     	|  9.083e-07      	|
|  [Strategy]SingleDeviceStrategy.on_validation_end                                                                                                                                                           	|  4.61e-06       	|  2              	|  9.2201e-06     	|  8.1223e-07     	|
|  [LightningModule]NeRFSystem.configure_sharded_model                                                                                                                                                        	|  9.0599e-06     	|  1              	|  9.0599e-06     	|  7.9811e-07     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_end                                                                                                                                                          	|  4.4303e-06     	|  2              	|  8.8606e-06     	|  7.8056e-07     	|
|  [Strategy]SingleDeviceStrategy.on_validation_start                                                                                                                                                         	|  4.3702e-06     	|  2              	|  8.7405e-06     	|  7.6997e-07     	|
|  [LightningModule]NeRFSystem.on_fit_end                                                                                                                                                                     	|  7.9302e-06     	|  1              	|  7.9302e-06     	|  6.986e-07      	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_start                                                                                                                                                        	|  6.8704e-06     	|  1              	|  6.8704e-06     	|  6.0523e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_start                                                                                                                                                     	|  6.3796e-06     	|  1              	|  6.3796e-06     	|  5.6199e-07     	|
|  [Callback]TQDMProgressBar.on_validation_epoch_start                                                                                                                                                        	|  3.0599e-06     	|  2              	|  6.1197e-06     	|  5.391e-07      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_start                     	|  5.8496e-06     	|  1              	|  5.8496e-06     	|  5.1531e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_end                                                                                                                                            	|  2.8252e-06     	|  2              	|  5.6503e-06     	|  4.9775e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_start                	|  2.7702e-06     	|  2              	|  5.5404e-06     	|  4.8807e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_end            	|  2.6901e-06     	|  2              	|  5.3803e-06     	|  4.7396e-07     	|
|  [LightningModule]NeRFSystem.on_validation_epoch_start                                                                                                                                                      	|  2.5094e-06     	|  2              	|  5.0189e-06     	|  4.4213e-07     	|
|  [Callback]TQDMProgressBar.on_fit_end                                                                                                                                                                       	|  4.5896e-06     	|  1              	|  4.5896e-06     	|  4.0431e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_validation_epoch_start          	|  2.2599e-06     	|  2              	|  4.5197e-06     	|  3.9815e-07     	|
|  [Callback]GradientAccumulationScheduler.on_train_end                                                                                                                                                       	|  4.5002e-06     	|  1              	|  4.5002e-06     	|  3.9643e-07     	|
|  [LightningModule]NeRFSystem.on_validation_end                                                                                                                                                              	|  2.245e-06      	|  2              	|  4.4899e-06     	|  3.9553e-07     	|
|  [Callback]TQDMProgressBar.on_fit_start                                                                                                                                                                     	|  4.4191e-06     	|  1              	|  4.4191e-06     	|  3.8929e-07     	|
|  [Callback]GradientAccumulationScheduler.on_validation_epoch_start                                                                                                                                          	|  2.2002e-06     	|  2              	|  4.4005e-06     	|  3.8765e-07     	|
|  [Callback]GradientAccumulationScheduler.setup                                                                                                                                                              	|  4.2403e-06     	|  1              	|  4.2403e-06     	|  3.7354e-07     	|
|  [Callback]TQDMProgressBar.on_configure_sharded_model                                                                                                                                                       	|  4.1788e-06     	|  1              	|  4.1788e-06     	|  3.6813e-07     	|
|  [Callback]TQDMProgressBar.on_before_accelerator_backend_setup                                                                                                                                              	|  3.54e-06       	|  1              	|  3.54e-06       	|  3.1185e-07     	|
|  [LightningModule]NeRFSystem.prepare_data                                                                                                                                                                   	|  3.2298e-06     	|  1              	|  3.2298e-06     	|  2.8453e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_start                                                                                                                                          	|  2.9895e-06     	|  1              	|  2.9895e-06     	|  2.6336e-07     	|
|  [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup                                                                                                                                	|  2.9607e-06     	|  1              	|  2.9607e-06     	|  2.6081e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_start                                                                                                                                                       	|  2.8685e-06     	|  1              	|  2.8685e-06     	|  2.5269e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_before_accelerator_backend_setup	|  2.8005e-06     	|  1              	|  2.8005e-06     	|  2.467e-07      	|
|  [Callback]TQDMProgressBar.teardown                                                                                                                                                                         	|  2.7297e-06     	|  1              	|  2.7297e-06     	|  2.4047e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_train_end                       	|  2.6589e-06     	|  1              	|  2.6589e-06     	|  2.3423e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_start          	|  2.6301e-06     	|  1              	|  2.6301e-06     	|  2.3169e-07     	|
|  [LightningModule]NeRFSystem.on_fit_start                                                                                                                                                                   	|  2.55e-06       	|  1              	|  2.55e-06       	|  2.2463e-07     	|
|  [Callback]TQDMProgressBar.on_pretrain_routine_end                                                                                                                                                          	|  2.5406e-06     	|  1              	|  2.5406e-06     	|  2.2381e-07     	|
|  [Callback]GradientAccumulationScheduler.on_configure_sharded_model                                                                                                                                         	|  2.5397e-06     	|  1              	|  2.5397e-06     	|  2.2373e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model         	|  2.5304e-06     	|  1              	|  2.5304e-06     	|  2.2291e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_start                       	|  2.5202e-06     	|  1              	|  2.5202e-06     	|  2.2201e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_start                                                                                                                                                      	|  2.1802e-06     	|  1              	|  2.1802e-06     	|  1.9206e-07     	|
|  [LightningModule]NeRFSystem.on_save_checkpoint                                                                                                                                                             	|  2.1197e-06     	|  1              	|  2.1197e-06     	|  1.8673e-07     	|
|  [Callback]GradientAccumulationScheduler.teardown                                                                                                                                                           	|  2.1094e-06     	|  1              	|  2.1094e-06     	|  1.8583e-07     	|
|  [Callback]GradientAccumulationScheduler.on_fit_end                                                                                                                                                         	|  2.1001e-06     	|  1              	|  2.1001e-06     	|  1.8501e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_fit_end                         	|  2.0694e-06     	|  1              	|  2.0694e-06     	|  1.823e-07      	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_pretrain_routine_end            	|  1.9297e-06     	|  1              	|  1.9297e-06     	|  1.6999e-07     	|
|  [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 20, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown                           	|  1.8794e-06     	|  1              	|  1.8794e-06     	|  1.6556e-07     	|
|  [Callback]GradientAccumulationScheduler.on_pretrain_routine_end                                                                                                                                            	|  1.8692e-06     	|  1              	|  1.8692e-06     	|  1.6466e-07     	|
|  [LightningModule]NeRFSystem.on_train_start                                                                                                                                                                 	|  1.6699e-06     	|  1              	|  1.6699e-06     	|  1.471e-07      	|
|  [LightningModule]NeRFSystem.on_train_end                                                                                                                                                                   	|  1.5507e-06     	|  1              	|  1.5507e-06     	|  1.366e-07      	|
|  [Strategy]SingleDeviceStrategy.on_train_start                                                                                                                                                              	|  1.4594e-06     	|  1              	|  1.4594e-06     	|  1.2856e-07     	|
|  [LightningModule]NeRFSystem.teardown                                                                                                                                                                       	|  1.4501e-06     	|  1              	|  1.4501e-06     	|  1.2774e-07     	|
|  [LightningModule]NeRFSystem.on_pretrain_routine_end                                                                                                                                                        	|  1.2005e-06     	|  1              	|  1.2005e-06     	|  1.0575e-07     	|
|  [Strategy]SingleDeviceStrategy.on_train_end                                                                                                                                                                	|  1.0505e-06     	|  1              	|  1.0505e-06     	|  9.2544e-08     	|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
