Skip to end of metadata
Go to start of metadata
The website has additional iRODS troubleshooting info their Troubleshooting Guide.



Is there a diagram somewhere of how this stuff is setup?
  • For NCMIR with CCDB, there is a diagram, here.
What directories store iRODS "stuff"?
  • /opt/irods
  • /etc/irods
  • /var/opt/irods
  • /var/run/irods
  • /etc/init.d
    . What files are most important in configuration and troubleshooting of iRODS?
  • /opt/irods
    • /server/bin/irodsctl (binary invoked by initscript)
    • /server/config/reConfig (iRODS rules)
  • /etc/irods
    • /<proj><env>irods (configuration files that tune the binary to a project and environment)
  • /var/opt/irods
    • /<proj><env>irods/log (log files)
  • /var/run/irods
    • /<proj><env> (process lock file - sticks around if irods dies messily)
How can I increase the logging level?
  • (iRODS 3.2) export spLogLevel=5, then restart iRODS
  • (iRODS 3.2) spLogLevel can be set by editing or setting the environment variable spLogLevel and then restarting the server. If set to "5", it is equal to LOG_NOTICE which will log messages of the 'notice' severity and more severe (lower numbers).
What decides the default resource?

 NOTE: defaultResource does not apply to irodsAdmin users

  • (iRODS 2.5) This and other rules are in the /opt/irods/server/config/reConfig/*.irb files.
  • (iRODS 3.2) This and other rules are in the /opt/irods/server/config/reConfig/*.re files.
Are the iRODS configuration files templated?
  • Yes. See modules/m_irods/templates
Where do the values for the templates get set?
  • (iRODS 2.5) In the project module. See modules/p_ccdb/manifests/init.pp for an example.
  • (iRODS 3.2) In the m_irods module. See modules/m_irods/manifests/params.pp and modules/m_irods/manifests/ccdb/params.pp for an example.
How do I find all the files the iCAT thinks are on a particular resource?
How do I move data from one resource to another?
  • In the example below:
    • permissions and ownership of the original data is changed
      • everything is owned by ccdbprodirods (aka rawdata)
      • ccdbprodirods (aka rawdata) has rw to everything
      • everything is group owned by gid=100 (aka neuro, aka users)
      • the group has read/write to anything in the montage processed_data reconstruction segmentation directories
    • any data already in iRODS is iphymv'd to the new resource, then
    • any files that have been generated, but not yet sync'd (i.e. cataloged in the iRODS database) are moved to the new resource in order to keep all the data associated with a given MPID together, then
    • any symbolic links that have been created are moved to the new resource
  • Icon

    Before running the next bit, any directories that are missing in the new location need to be created

I'm getting "rcAuthResponse failed with error -827000 CAT_INVALID_USER" in the log. What's wrong?
  • The user isn't in the iCAT. You may have misspelled the name, forgotten the extension, used the wrong zone or connected to the wrong environment.
I'm getting "rcAuthResponse failed with error -826000 CAT_INVALID_AUTHENTICATION" in the log. What's wrong?
  • The password you entered is incorrect.
I'm getting "ERROR: rmUtil: rm error for /telescience/home/CCDB_DATA_USER.portal/irmCDUunreg, status = -528002 status = -528002 UNIX_FILE_RENAME_ERR, No such file or directory" What's wrong?
  • Check the permissions on the disk. If you don't have permission to the unix file, it acts like the file isn't there.
I'm getting "iadmin rmresc ucsd-ccdb-prod-nas5 ; ERROR: Level 0: resource 'ucsd-ccdb-prod-nas5' contains one or more dataObjects ; ERROR: rcGeneralAdmin failed with error -835000 CAT_RESOURCE_NOT_EMPTY". What's wrong?
  • Use the iquest command above for finding what the iCAT thinks is on a particular resource.
I'm getting "ERROR: rmUtil: rm error for /telescience/home/ccdbdevirods/irodsupgrade, status = -13000 status = -13000 SYS_NO_API_PRIV" when I try to 'irm' a file that I own. What's wrong?
  • In iRODS 3.2, the ACL on the top level collection is restricted to the admin user only. Use the -f flag on the 'irm' command, or modify the ACL on the trash collection.
I'm getting "ERROR: connectToRhost: error returned from host status = -38000 status = -38000 SYS_AGENT_INIT_ERR ; ERROR: _rcConnect: connectToRhost error, server on is probably down status = -38000 SYS_AGENT_INIT_ERR" when I run 'ils' on the iCAT and it looks like the reagent is not starting because 'ps ax' has "10620 ? Ssl 0:00 ./irodsServer PROJECT=ccdbdevirods ; 10629 ? Z 0:00 [irodsReServer] <defunct>" What's wrong?
  • The reAgent isn't starting. Check both the rodsLog and the reLog for info. It was a parsing problem with the file in this case.
" when I try to 'irm' a file that I own. What's wrong?
  • In iRODS 3.2, the ACL on the top level collection is restricted to the admin user only. Use the -f flag on the 'irm' command, or modify the ACL on the trash collection.
Only rods user can write to dev1 and only rodsadmin can write to dev0. What's wrong?
  • I moved the dev0 vault from the slave to the iCAT using the 'iadmin modresc' command. When I moved the vault back to the slave, the problem went away.
I'm trying to do Resource Monitoring. How the heck do you get this to work?
From Slack message 2017



What does the process list look like when I use scp to copy something to iRODS via fuse?

 It looks like the iCAT can't talk to the Oracle database. What should I do?

If you see the following in the /var/opt/irods/ccdb<env>irods/log/rodslog... file:

Try this:

or this

If that doesn't work, check the status of the database server, including checking disk space, load, and that the database engine is running.


I want to Archive data that isn't in the data jail. What should I do?

On the

  • mv the directory that contains the data to the appropriate place in the data jail and rename the directory to include the prefix "CCDBID_<MPID>_"
    • e.g.

  • recursively change the ownership of the data
    • e.g. chown -R rawdata:users CCDBID_<MPID>_<DATA_DESC>

The Archive process initiated from the CCDB Portal was interrupted. What should I do?
  • FASTER OPTION - Use if no data made it to the final archive destination
    • Clear out the temporary storage space in the archive

    • Remove any <MPID_directory> data from the archive

      Archive Rollback Cautions


      Only do this if you're SURE it's safe to delete everything under the <MPID_directory> (It should be either empty, or have only data for which a copy still exists in the data jail. The next command assumes all rawdata in the archive for that MPID can be DELETED. All rawdata for that <MPID_directory> must be in the data jail before using this procedure.

    • "Archive" the data again from the CCDB Portal


  • SAFER OPTION - Use if some, but not all data made it to the final archive destination
    • md5sum each file in the data jail and compare to md5sum for corresponding file in final archive destination

    • If the md5sums do not match, correct it so that the correct file is in both places.
    • Do an iscan on the archive directory to see if any files were registered with iRODS, but which never made it to the right place on the disk.


      If files show up as "orphaned" (meaning it's on the disk, but not cataloged in iRODs), they need to be "sync"d via the CCDB Portal.


      An ils of an orphaned file will says it doesn't exist in iRODS.

    • Copy any missing files from the data jail to the final archive location

      Copy Caution


      Be careful not to overwrite data you don't want to be overwritten - copy only the missing files, or try using 'cp -i ...'

    • "sync" the archive directory via the CCDB Portal.
      The portal will send email telling you the data has been synced.

    • Rerun the iscan to verify it runs clean.

How do I know if the service started properly?
  • ON AN ICAT you'll see both an irodsServer ANDand irodsReServer process running. The output of `ps ax | grep irod` should look like this:

  • ON A SLAVE you'll see onlyan irodsServer process running. The output of `ps ax | grep irod` should look like this:








  • fugazi, crocket and lakota are all icats
  • irods logs are in /var/opt/irods/ccdb<env>irods/log/*
  • the pid is in /var/run/ccdb<env>
  • the exectuables are in /opt/irods/
  • the configuration files are in /etc/irods/ccdb<env>irods


  • No labels