| Author |
 Topic  |
|
|
ark19
Junior Member
27 Posts |
 Posted - Jan 31 2012 : 1:15:34 PM
|
Hi everyone,
I'm having some trouble understanding why some of my cluster jobs are taking so long, and I'm hoping someone could shed some light on what's going on.
My confusion stems from the way I've interpreted these observations:
-if I submit a single subject, the job will consistently finish in ~5 hours
-I submitted a batch of 5 subjects in a single job, and it took ~2 days to run
-I simultaneously submitted 30 batches of 10 subjects/job, and they all took ~6 days to run
I decided then to submit the remaining subjects individually but simultaneously, hoping that since they'd be processing in parallel, they would all finish in certainly fewer than 10 hours. However, at this point they've been running for over 15 hours. The job I'm running is first-level processing with the conn toolbox for SPM/Matlab. Could anyone tell me what's missing from my understanding of how these processing times should come out?
Thanks,
Annchen |
|
|
petty
BIAC Staff
USA
453 Posts |
Posted - Jan 31 2012 : 2:35:38 PM
|
Things are obviously going to depend on other process/jobs/users and overall cluster usage.
However we've recently found that there are some serious performance impacts when the filesystem you are using reaches a certain capacity. BlueArc gives serious warnings about going over 90%, with 80% being the max suggested.
In your case, the Hariri system is at 95% full. What this means is that anytime the fileserver has to search for a block to write out data, its going to have to search much longer to find the appropriate space ... therefore times are going to creep up.
Clearing up space is going to help if you can get it down far enough. Having Francis remove a couple of snapshots as well. |
 |
|
|
ark19
Junior Member
27 Posts |
Posted - Jan 31 2012 : 7:03:28 PM
|
| Many thanks - I had my suspicions but wanted to hear it from somebody who knew. We'll do our best to address these issues! |
|
|
| |
Topic |
|
|
|
cluster processing time
