Natural language processing in tensorflow
Week 1 A simple intro to the Keras Tokenizer API ```python from tensorflow.keras.preprocessing.text import Tokenizer
Posts by Year
2021
Nv dli deep learning at scale with horovod
Steps to implement Horovod Initialize Horovod and Select the GPU to Run On Print Verbose Logs Only on the First Worker Add Distributed Optimizer Ini...
Nv dli fundamentals of deep learning for multi Gpus lab2
Lab 2: Multi-GPU DL Training Implementation using Horovod Horovod is a distributed deep learning training framework. It is available for TensorFlow, Keras, P...
Nv dli fundamentals of deep learning for multi Gpus lab1
Lab 1: Gradient Descent vs Stochastic Gradient Descent, and the Effects of Batch Size Gradient Descent ```python #Generating a random dataset Numpy is a fund...
Nv dli fundamentals of dl
At times you will need to clear the GPUs memory, either to reset the GPU state when an experiment goes wrong, or, in between notebooks when you need a fresh ...
Launching Crossnode Pytorch Jobs with Mpi
Running these via mpirun or srun like on https://developer.nvidia.com/blog/how-to-run-ngc-deep-learning-containers-with-singularity/ would help to ensure cor...
Gpu utilization check for multinode slurm job
Get a snapshot of GPU stats without DCGM.
2020
Lammps compilation on nersc
NOTE: NERSC provides compiled LAMMPS binaries as modules. This tutorial is for users with modified source code.
Download nimh data with ndatools
In order to access NIMH Data Archive, users need to apply for an account using this link. Project PI needs to add the new user to their project for data acce...
Create R environment using conda
```bash #install Miniconda mkdir ~/conda; cd ~/conda wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh bash Miniconda3-latest-Linux-...
Windows using conda environments in spyder
In Anaconda Prompt conda create -n newEnv conda activate newEnv - y conda install spyder-kernels -y conda install ... where python Copy the Python environme...
Rstudio installation with conda
Install Miniconda if you haven’t done so. mkdir ~/conda; cd ~/conda wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh bash Mi...
R jupyter notebook kernal
Install Miniconda and configure Jupyter notebook to run R kernals. First make sure X11 forwarding works on your local machine. Then follow the steps below. `...
Zfs check uncompressed file size
Use the GNU du command [jingchao@login.crane hao]$ du --apparent-size -sh 11T . [jingchao@login.crane hao]$ du -sh 3.2T .
Aepera quick setup
Aspera is IBM’s high-performance file transfer software which allows for the transfer large files and data sets with predictable, reliable and secure deliver...
2019
Ms word replace line break with return
This is mostly used for preparing a reponse letter for journal reviews. Copying/pasting reviewers’ comments to a MS Word file ends up with lots of line break...
Lsdyna mpp submission script with slurm
```bash #!/bin/sh Change the job name, if you want to #SBATCH –job-name=pendulum
Matlab parallel server
This document provides the steps to configure MATLAB to submit jobs to a cluster, retrieve results, and debug errors.
Check whether infiniband is being used
Check for number incrementals in files at /sys/class/infiniband/mlx4_0/ports/1/counters/ watch -n .5 -diff=cumm 'grep -H . /sys/class/infiniband/mlx4_0/ports...
Matlab alexnet support package install
If Deep Learning Toolbox Model for AlexNet Network support package is not installed, then the function provides a link to the required support package in the...
Create shared conda environment
create a shared location in $COMMON mkdir /common/GROUP/conda chown :GROUP conda chmod 770 conda create conda env in the shared location ...
Slurm_cgroup_memory_usage
Put the line below at the end of a slurm submit script to find out the memory usage. cgget -g memory /slurm/uid_${UID}/job_${SLURM_JOB_ID}
User lockout
Global lock. usermod.py --lock user
2018
Coldfront deployment
install LDAP dependences yum install python-devel openldap-devel -y pip install django-auth-ldap pip install python-ldap ldap3
Anvil modify instance offline
```console [root@beta ~]# source keystonerc_admin
Notes on hcc purge
Crane/Tusker purged files are temporarily stored at /lustre/backup/robinhood/YYYY-WXX (YYYY=year and XX=week number) for 2 weeks. Files beyond 2 weeks are...
Lammps intel compiler openmpi compilation
wget https://github.com/lammps/lammps/archive/stable_22Aug2018.tar.gz tar zxvf stable_22Aug2018.tar.gz cd lammps-stable_22Aug2018/src for i in BODY ...
Smileimpi compilation
Compile OpenMPI with MPI_THREAD_MULTIPLE support. ./configure --enable-mpi-thread-multiple ml load compiler/gcc/4.9 openmpi/1.10 phdf5/1.8 python/3.6 ...
Anvil add new user
From Tusker/Crane, ssh beta.anvil.hcc.unl.edu sudo su - source /root/keystonerc_admin /util/accounts-mgmt-cli/anvilman.py -v -g GROUPNAME -u USERNAM...
Cp2k compilation with plumed
Building Your Own FFTW3 Interface Wrapper Library source /util/opt/lmod/lmod/init/profile ml load compiler/intel/16 cd /util/comp/intel/2016.3/mkl/interfaces...
Acs reviewing certificate
Click the button to see and download your peer review activity certificate. We hope you will display it with the same pride we take in working with such an a...
My elsevier reviews profile
Peer review is the cornerstone of science, and Elsevier is dedicated to supporting and recognizing our journals’ reviewers. My Elsevier Reviews Profile aims ...
Siesta v4.1.b3 openmpi compilation with slurm example
SIESTA is both a method and its computer program implementation, to perform efficient electronic structure calculations and ab initio molecular dynamics simu...
El 7 docker upgrade to docker Ce
yum erase *docker* -y yum-config-manager –add-repo “https://download.docker.com/linux/centos/docker-ce.repo” yum install docker-ci -y systemctl stop...
Module usage search in splunk
https://slurp.unl.edu:8000/en-US/account/login?return_to=%2Fen-US%2Fapp%2Flauncher%2Fhome HCC username and password “Launch search app” “Lmod loggin...
Conda build
Tensorflow 1.10.1 https://www.tensorflow.org/install/source#tested_source_configurations https://github.com/tensorflow/tensorflow/blob/v1.10.1/tensorflow/too...
Hcc Puppet
https://hcc-git.unl.edu/red-puppet.git git clone git@hcc-git.unl.edu:red-puppet
Jupyter Notebook Customized Kernels
```bash module load anaconda conda create -n tensorflow-keras tensorflow keras ipykernel python=3.6
Conda project
New Project ```bash mkdir packages/blast cd packages/blast cp ../../anaconda-project.yml.example ./anaconda-project.yml anaconda-project add-env-spec -n blas...
Octopus intel Mkl openmpi
Octopus FFTW3 OpenMPI Intel-MKL GSL
Boost openmpi
Boost $ ./bootstrap.sh --prefix=/home/username/usr $ vim boost_1_55_0/tools/build/v2/user-config.jam using mpi ；#add this to the last line $ ./b2 install --p...
Xdmod boot from snapshot
OpenStack Flavor: general-xlarge Boot from snapshot: “hcc-xdmod prod snap 3/8/18” Security Groups: Port 8080/tcp; default Net...
Slurm cluster usage per group
$ sreport cluster AccountUtilizationByUser accounts=swanson start=1/1/17 end=12/31/17 -t Hour ---------------------------------------------------------------...
Lustre Acl Control
get the ACL’s for /home/usera [usera@login-damiana ~]# lfs lgetfacl /home/usera # file: home/usera # owner: usera # group: users user::rwx group::--- other:...
2017
Slurm Setup
OS is CentOS 7
Comsol v5.3 linux server
Problem: VM run out of entropy cat /proc/sys/kernel/random/entropy_avail 13
R package installations
To see the directories that R searches for libraries, from within R you can do: .libPaths();
Xalt server setup
OpenStack Manage Security Group Rules: XALT
Time sync using chrony
chrony is a versatile implementation of the Network Time Protocol (NTP). It can synchronize the system clock with NTP servers, reference clocks (e.g. GPS rec...
Apache ssl
xdmod.conf ```bash Redirect HTTP to HTTPS <VirtualHost *:80> ServerName hcc-xdmod.unl.edu Redirect / https://hcc-xdmod.unl.edu </VirtualHost&g...
Xdmod fresh install
yum install epel-release -y yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm -y yum groupinstall "Development Tools" -y yum...
Xdmod purge
```bash mysql Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 1299 Server version: 5.5.50-MariaDB MariaDB Server
2016
Install Perl Module In User Directory
option 1 echo 'eval $(perl -Mlocal::lib)' >> ~/.bashrc source ~/.bashrc cpan cpan> install Module::Load::Conditional
Globus Connect Cli File Transfer
Add ssh key to your Globus account. instruction ssh jingchao@cli.globusonline.org $ transfer -- hcc#tusker/home/swanson/jingchao/file1 hcc#crane/home/sw...
Rmpi Cluster Installation
module load compiler/gcc openmpi R export R_LIBS=$HOME/R/x86_64-pc-linux-gnu-library/3.3 mkdir -p $HOME/R/x86_64-pc-linux-gnu-library/3.3 wget https://cran.r...
Enable Ldap Centos7 Vm
Only essential commands are listed. Confidential info varies by sites.
Duo Integration With Xdmod Simplesamlphp
To reduce noise, turn off SElinux and firewall. Clone the simplesamlphp-duosecurity repo and put it in /usr/share/xdmod/vendor/simplesamlphp/simplesamlp...
Xdmod Multiple Clusters Ingestor
Setup daily cron job on each cluster running command: