This page looks best with JavaScript enabled

A list of *NIX commands

 ·  ☕ 15 min read  ·  ✍️ Alexandru Munteanu

Table of Contents

  1. File Permissions
    1. mounting a drive with read and write permissions for specific users
  2. The apt package manager on Debian-based systems
    1. search a package in the repository
    2. list what packages a package depends on
    3. held-back packages
  3. SystemD
    1. Show a list of SystemD timers
    2. Status of a SystemD process
    3. List all the SystemD unit files
    4. journalctl
      1. Show the usage of SystemD logs
      2. Remove SystemD logs that are older than a period of time
  4. Mounting
    1. List PID’s of processes that work with a certain mountpoint
  5. Filesystem
    1. Symbolic and hard links
  6. Misc
    1. Show the filesize and filename sorted by filesize
    2. find out who is logged in and what they are doing
    3. Remove a user from a group
    4. Sort files
      1. Sort files based on their filename which includes numbers (versions):
  7. File Compression
    1. Create a .tar.gz archive with the gzip -9 (best compression)
  8. The find tool
    1. Find and sort by size all vi swp files
    2. Find and remove
    3. Move files that are bigger in size than X
  9. Virtualization
    1. VirtualBox VBoxManage
      1. Start a virtual machine:
      2. Power off a virtual machine:
      3. List all running VM’s
    2. Virsh
      1. Installing with virsh
    3. kvm
      1. Start a virtual machine with kvm
  10. The grep tool
    1. Search for text in a whole directory’s files
  11. emacs keybindings in bash
  12. Monitoring progress when writing with dd
    1. verbose using status=progress
    2. using pv
  13. portage
    1. Whenever the system complains about zz-autounmask.
    2. emerge
      1. update packages
      2. package information
    3. updating packages
    4. get USE flag description
    5. check how long a package took to compile (emerge)
    6. see current emerging packages
    7. list of USE flags
    8. USE flags db
    9. multiple packages within a single pkg slot
  14. CUPS
    1. localhost:631 -> cups web interface
    2. list installed printers and their status
    3. Available options
  15. tmux
    1. new session with name
    2. list sessions
    3. attach session
    4. shortcuts
      1. Sessions
      2. Windows (tabs)
      3. Panes (splits)
    5. navigation
    6. saving buffers in tmux
  16. networking
    1. openWRT
      1. Kick a client (with optional ban_time)
    2. DNS
      1. checks if the resolver and dns is working correctly
      2. dig query DNS records
    3. iwlist
      1. get all access points:
    4. ARP table
      1. List the ARP table:
      2. updating the arp table
    5. Request DHCP lease
    6. nmap
      1. Probe open ports to determine service/version info
    7. openvpn
      1. make a persistent tunnel
      2. connect to openvpn
      3. delete a tun interface
    8. list of well-known ports:
    9. ip
      1. ip route to find the gateway
      2. rename interface
      3. check if there’s a connection
    10. iptables
      1. list line numbers
      2. delete iptable entry
    11. netcat for video file streaming
    12. nmcli
      1. list of available wifi networks:
      2. list all configured networks
      3. connect to a configured network with nmcli
  17. docker
    1. Copy files to/from a docker container
    2. list docker images
    3. force remove an image
    4. run command in docker container
      1. Example (start a shell):
    5. Cleans the whole system
  18. bash scripts shenanigans
    1. Remove every 2nd file in current dir
  19. ImageMagick
    1. Convert a directory of images to PDF
  20. Weechat
  21. The Git version system
    1. revert a file state back to an earlier commit
      1. example:
    2. merge another branch into the current one
  22. Uncategorised
    1. Run a command at a certain time
      1. example
    2. find out memory information (number of ram slots, size, max supported, etc)
    3. find out library dependencies of some software
      1. Example:
  23. freeBSD stuff
    1. list all disks (equivalent of lsblk)
  24. urxvt-perls
  25. miller
    1. Print a csv that into the table format
  26. ulimit
    1. change max file descriptors to 4096:
  27. arch stuff
    1. Keep your Arch Linux USB drive handy
    2. Pacman Tips & Tricks
      1. DO NOT clear the Pacman cache, ever!
      2. List all packages and their size on disk
  28. check what two files have in common
  29. android stuff
    1. mounting android storage over sshfs
  30. ssh
    1. ssh with X11 forwarding, a faster way with compression
  31. The mmv tool - Mass move
  32. Xorg and Input methods
    1. enable moving cursor with numpad keys
    2. synclient (synaptics touchpad) deactivate touchpad tap
  33. partioning
  34. Geospatial data processing tools
    1. ogr2ogr
      1. Conversion of OpenFileGDB to ESRI Shapefile
    2. gdal
      1. merge multiple rasters into one multi-band raster using gdal_mereg
      2. rasterization (burning vectors to rasters) using gdal_rasterize
    3. ESA-SNAP
      1. List of SNAP modules
      2. Reprojection of S3 Sentinel archives using ESA GPT
      3. Snap.parallelism
  35. MongoDB
    1. Import data to a mongo database
    2. Export data from a mongo database
    3. Create user relative to database:
  36. zsh
    1. Printing a file without it’s extension
  37. fzf bindings

File Permissions

A good way to remember permisions:

r = 4
w = 2
x = 1

You can sum them up, for exameple rxw = 4 + 2 + 1 = 7, or rx = 4 + 1 = 5

mounting a drive with read and write permissions for specific users

pi@cyberdelia /storage % sudo mount /dev/sda1 storage/ -o umask=0022,gid=1000,uid=1000

The apt package manager on Debian-based systems

search a package in the repository

apt-cache search <pkgname>

list what packages a package depends on

apt-cache depends <pkgname>

held-back packages

If a package changes dependencies it won’t automatically upgrade anymore

To upgrade:

apt-get --with-new-pkgs upgrade

SystemD

Show a list of SystemD timers

systemctl list-timers

Status of a SystemD process

systemctl status processname

List all the SystemD unit files

systemctl list-unit-files --all

journalctl

Show the usage of SystemD logs

journalctl --disk-usage

Remove SystemD logs that are older than a period of time

In this case, logs older than 2 days will be deleted:

journalctl --vacuum-time=2d

Mounting

List PID’s of processes that work with a certain mountpoint

fuser -c <mountpoint>

Filesystem

man hier - description of the filesystem hierarchy

ln file link -> hard link named link to file
ln -s file link -> soft link named link to file
ls -i -> list inode number

Misc

Show the filesize and filename sorted by filesize

du -sh * | sort -h

find out who is logged in and what they are doing

w (w is the actual command name)

Remove a user from a group

gpasswd -d <username> <group>

Sort files

Sort files based on their filename which includes numbers (versions):

sort --version-sort

alex@beast ~/thesis-presentation/video % ls *.mp3 | sort --version-sort

output_MP3WRAP.mp3
slide1.mp3
slide2.mp3
slide3.mp3
slide4.mp3
slide5.mp3
slide6.mp3
slide7.mp3
slide8.mp3
slide9.mp3
slide10.mp3
slide11.mp3
slide12.mp3
slide13.mp3
slide14.mp3
slide15.mp3
slide16.mp3
slide17.mp3
slide18.mp3
slide19.mp3
slide20.mp3
slide21.mp3
slide22.mp3

File Compression

Create a .tar.gz archive with the gzip -9 (best compression)

tar -I "gzip --best" -c -f archive.tar.gz dirname

The find tool

Find and sort by size all vi swp files

find . -type f -name "*.swp" | xargs du -sh | sort -h

Find and remove

find . -type f -name "*.swp" -exec rm -rf {} \;

Move files that are bigger in size than X

This command will move files that are bigger than 10MB:

find . -type f -size +10M -exec mv "{}" new\_folder/ \;

Virtualization

VirtualBox VBoxManage

Start a virtual machine:

VBoxManage startvm <wm-name> --type headless

Power off a virtual machine:

vboxmanage controlvm <wm-name> poweroff

List all running VM’s

VBoxManage list runningvms

Virsh

Installing with virsh

virt-install \
-n FreeDOS \
--description "FreeDOS" \
--os-type=windows \
--os-variant=msdos \
--ram=2048 \
--vcpus=2 \
--disk path=/home/alex/qemu/freedos/freedos.img,bus=virtio,size=4 \
--graphics none \
--cdrom /home/alex/Downloads/FD12FULL.img \
--check path_in_use=off

kvm

Start a virtual machine with kvm

kvm -hda freebsd.img -m 1024 -net nic -net user -soundhw all &!

The grep tool

Search for text in a whole directory’s files

grep -rnw . -e 'LED_ON'

emacs keybindings in bash

CTRL-P go to the Previous command in your history
CTRL-N go to the Next command in your history
CTRL-R Reverse-search through your history
CTRL-S Search forward through your history
CTRL-A Move the cursor to the beginning of the line
CTRL-E Move the cursor to the end of the line
CTRL-W delete a Word backwards
ALT-D delete a word forwards
CTRL-F move the cursor Forward 1 character
CTRL-B move the cursor Backward 1 character
ALT-F move the cursor Forward 1 word
ALT-B move the cursor Backward 1 word
ALT-_ undo

Monitoring progress when writing with dd

By default, dd does not output progress on how much it has already written.
There are two ways to get the status of dd: either by the status=progress arugment, or by piping through the pv tool.

verbose using status=progress

sudo dd if=livedvd-amd64-multilib-20160704.iso of=/dev/sdc bs=1M status=progress

using pv

sudo dd if=livedvd-amd64-multilib-20160704.iso | pv | of=/dev/sdc bs=1M

portage

Whenever the system complains about zz-autounmask.

dispatch-conf

emerge

update packages

after setting/modifying a package’s use flag in /etc/portage/package.use/pkg:

emerge -auDvN world

Everything in need of a recompilation will be emerged again.

package information

emerge --info <package>

updating packages

emerge -auDNv world
dispatch-conf
revdep-rebuild -i
eclean-dist -id

get USE flag description

euses -i <use-flag>

check how long a package took to compile (emerge)

genlop -t <package>

see current emerging packages

genlop -c

list of USE flags

less /var/db/repos/gentoo/profiles/use

USE flags db

less /var/db/repos/gentoo/profiles/use.desc

multiple packages within a single pkg slot

!!! Multiple package instances within a single package slot have been pulled
!!! into the dependency graph, resulting in a slot conflict:

solve with --oneshot =pkg

CUPS

localhost:631 -> cups web interface

list installed printers and their status

lpstat -t

Available options

lpoptions -l

tmux

new session with name

tmux new -S <name>

list sessions

tmux list-sessions

attach session

tmux attach -t <session_name>

shortcuts

In tmux, hit the prefix ctrl+b (my modified prefix is ctrl+a) and then:

Sessions

:new new session
s list sessions
$ name session

Windows (tabs)

c create window
w list windows
n next window
p previous window
f find window
, name window
& kill window

Panes (splits)

% vertical split
" horizontal split
o swap panes
q show pane numbers
x kill pane

  • break pane into window (e.g. to select text by mouse to copy)
  • restore pane from window

⍽ space - toggle between layouts
q (Show pane numbers, when the numbers show up type the key to goto that pane)
{ (Move the current pane left)
} (Move the current pane right)
z toggle pane zoom

to enable navigation with up/down arrow keys: Ctrl+b [
to disable it press q

saving buffers in tmux

buffers in tmux can be saved to files:

tmux list-buffers
tmux save-buffer foo.txt

networking

openWRT

Kick a client (with optional ban_time)

ubus call hostapd.wlan0 del_client "{'addr':'$MAC', 'reason':5, 'deauth':false, 'ban_time':0}"

DNS

checks if the resolver and dns is working correctly

getent

dig query DNS records

Example (for all records, default are A (ipv4) records):

dig uvt.ro ANY

iwlist

get all access points:

sudo iwlist wlp2s0 scanning

ARP table

List the ARP table:

arp -a

updating the arp table

for i in {1..254}; do ping -c 1 192.168.178.$i & done

Request DHCP lease

dhcpcd

nmap

Probe open ports to determine service/version info

nmap -sV <host>

openvpn

make a persistent tunnel

openvpn --mktun --dev tun

Mon Aug 6 19:32:41 2018 TUN/TAP device tun1 opened
Mon Aug 6 19:32:41 2018 Persist state set to: ON

connect to openvpn

sudo openvpn --dev tunX --tls-client --config alex.ovpn

delete a tun interface

sudo openvpn --rmtun --dev tun0

list of well-known ports:

less /etc/services

ip

ip route to find the gateway

ip route | grep default

rename interface

ip link set wlan0 name wlp2s0

check if there’s a connection

ip link

iptables

list line numbers

sudo iptables -L --line-numbers

delete iptable entry

sudo iptables -D INPUT <linenr>

netcat for video file streaming

netcat -l -p 8111 < mymovie.mp4

You can then use mpv to play it like this:

netcat 192.168.0.250 8111 | mpv -

nmcli

list of available wifi networks:

nmcli dev wifi

list all configured networks

nmcli con

connect to a configured network with nmcli

nmcli con up <connection-name>

docker

Copy files to/from a docker container

By using docker cp:

$ docker cp foo.txt mycontainer:/foo.txt
$ docker cp mycontainer:/foo.txt foo.txt

list docker images

docker image ls

force remove an image

docker image rm --force

run command in docker container

docker run -it <container> <command>

Example (start a shell):

docker run-it ubuntu bash

Cleans the whole system

This deletes /var/lib/docker/overlay2

docker system prune -af

bash scripts shenanigans

Remove every 2nd file in current dir

#!/bin/bash

for file in `find . -type f | awk 'NR % 2 == 0'`; do
    rm $file  
done

ImageMagick

Convert a directory of images to PDF

convert * output.pdf

Or convert a single image to PDF

convert image.png image.pdf

Weechat

ctrl+h - window left
ctrl+j - window down
ctrl+k - window up
ctrl+l - window right
ctrl+z - window zoom (make current small window full screen)
ctrl+n - next buffer
ctrl+p - previous buffer
ctrl+y - clipboard paste
ctrl+r - search
ctrl+j - stop search

/window splitv -> split vertically
/window splith -> split horizontally

The Git version system

revert a file state back to an earlier commit

git checkout <commit-hash> -- <path_to_file>

example:

git checkout 6d19add -- src/spacenet/tools/create_gti_masks.py

merge another branch into the current one

git merge origin/<other-branch>

Uncategorised

Run a command at a certain time

at <time> 
command
ctrl+D

example

at 22:30
touch test.txt

find out memory information (number of ram slots, size, max supported, etc)

dmidecode -t memory

find out library dependencies of some software

ldd /usr/bin/package

Example:

alex@MicroWave ~ % ldd /usr/bin/urxvt
linux-vdso.so.1 (0x00007ffeaa4ff000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd022b93000)
libfontconfig.so.1 => /usr/lib/x86_64-linux-gnu/libfontconfig.so.1 (0x00007fd022b4d000)
libfreetype.so.6 => /usr/lib/x86_64-linux-gnu/libfreetype.so.6 (0x00007fd022a91000)
libXft.so.2 => /usr/lib/x86_64-linux-gnu/libXft.so.2 (0x00007fd02287b000)
libXrender.so.1 => /usr/lib/x86_64-linux-gnu/libXrender.so.1 (0x00007fd022671000)
libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6 (0x00007fd022530000)
libgdk_pixbuf-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgdk_pixbuf-2.0.so.0 (0x00007fd022508000)
libgobject-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0 (0x00007fd0224b2000)
libglib-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0 (0x00007fd022393000)
libstartup-notification-1.so.0 => /usr/lib/x86_64-linux-gnu/libstartup-notification-1.so.0 (0x00007fd022387000)
libperl.so.5.28 => /usr/lib/x86_64-linux-gnu/libperl.so.5.28 (0x00007fd02206a000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd022065000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd022042000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd021e81000)
/lib64/ld-linux-x86-64.so.2 (0x00007fd022e93000)
libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007fd021e47000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fd021e2d000)
libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007fd021df0000)
libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007fd021de7000)
libpng16.so.16 => /usr/lib/x86_64-linux-gnu/libpng16.so.16 (0x00007fd021dac000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fd021b8e000)
libxcb.so.1 => /usr/lib/x86_64-linux-gnu/libxcb.so.1 (0x00007fd021b64000)
libgmodule-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgmodule-2.0.so.0 (0x00007fd021b5e000)
libgio-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0 (0x00007fd0219a0000)
libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007fd021996000)
libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007fd021920000)
libxcb-util.so.0 => /usr/lib/x86_64-linux-gnu/libxcb-util.so.0 (0x00007fd021719000)
libX11-xcb.so.1 => /usr/lib/x86_64-linux-gnu/libX11-xcb.so.1 (0x00007fd021714000)
libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6 (0x00007fd021510000)
libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007fd02130a000)
libmount.so.1 => /lib/x86_64-linux-gnu/libmount.so.1 (0x00007fd0212a9000)
libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007fd021081000)
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007fd021067000)
libbsd.so.0 => /lib/x86_64-linux-gnu/libbsd.so.0 (0x00007fd020e50000)
libblkid.so.1 => /lib/x86_64-linux-gnu/libblkid.so.1 (0x00007fd020dfb000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fd020def

freeBSD stuff

list all disks (equivalent of lsblk)

geom disk list

or:

camcontrol devlist

urxvt-perls

Use Meta-U to activate url-select and then the vi keybindings to navigate

miller

mlr --icsv --opprint --barred cat logging-unetv14_spacenetroads_vegas_sgd_dice-thor.sage.ieat.ro.csv

ulimit

change max file descriptors to 4096:

ulimit -sn 4096

arch stuff

Keep your Arch Linux USB drive handy

Software updates rarely make your machine un-bootable. But when that happens, you need the live USB of Arch Linux to save yourself.
After booting to the live USB, mount your hard drive, arch-chroot into the root partition of your installation, then fix the problems.
For instance, you can boot the machine with the live USB and reverse software updates

Pacman Tips & Tricks

DO NOT clear the Pacman cache, ever!

Pacman, the package manager, stores all the packages you’ve installed, including older and not-in-use packages, in /var/cache/pacman/pkg.
Unless you tell Pacman to remove unused packages, those packages stay on your hard drive, occupying ever bigger portion of your storage space as time goes by.

But clearing the cache is not a good idea unless you know with absolute certainty that you don’t need them ever again.
When one of the packages breaks something after an update, you may want to rollback the update, and that’s when pacman cache comes to your rescue.
For example, if you’ve just upgraded your kernel, and you soon realise it doesn’t work, the simplest solution is to downgrade the kernel back to
the previous version. Inside /var/cache/pacman/pkg, you will find the tarball for your old kernel (e.g.: linux-4.7.6-1-x86_64.pkg.tar.xz).

To downgrade the kernel, simply use the following command:

sudo pacman -U /var/cache/pacman/pkg/linux-4.7.6-1-x86_64.pkg.tar.xz

Instead of keeping everything or deleting everything from the cache, consider running

paccache -rk 3

to keep three most recent versions of all the packages, and paccache -ruk 0 to remove uninstalled packages from the cache

List all packages and their size on disk

LC_ALL=C pacman -Qi | awk '/^Name/{name=$3} /^Installed Size/{print $4$5, name}' | sort -h

check what two files have in common

comm is the reverse of diff!

tardis@thor:/home/alex/datasets$ comm -1 -2 vegas_test_f vegas_train_f

android stuff

mounting android storage over sshfs

I’ve found all of the MTP implementations to be wanting in some way. Then
I discovered SSHelper, an SSH server for Android. Now I can mount my
phone’s storage with sshfs using a command like

sshfs -p 2222 192.168.1.144:/storage/emulated/0 /mnt/phone

ssh

ssh with X11 forwarding, a faster way with compression

ssh -X -C -c blowfish-cbc,arcfour alex@thor

The mmv tool - Mass move

Move all files with the structure spacenetroads_AOI_2_Vegas_img603.geojson to spacenetroads_AOI_2_Vegas_GT_img973.geojson:

mmv '*_*_*_*_*.geojson' '#1_#2_#3_#4_GT_#5.geojson'

Xorg and Input methods

enable moving cursor with numpad keys

setxkbmap -option keypad:pointerkeys

Use shift + numlock to activate afterwards

synclient (synaptics touchpad) deactivate touchpad tap

synclient TapButton1=1

partioning

Use gdisk instead of fdisk whenever you want to use GPT!

Geospatial data processing tools

ogr2ogr

Conversion of OpenFileGDB to ESRI Shapefile

An example using CLC:

ogr2ogr -f “ESRI Shapefile” clc.shp /data/syno1/sage-storage/forestry-segmentation/corine/CLC2018_CLC2018_V2018_20.gdb

gdal

merge multiple rasters into one multi-band raster using gdal_mereg

gdal_merge.py -separate -of GTiff -o T35TMN_20180824T090549_B03_10_RGBNIR.tif T35TMN_20180824T090549_B02_10_IN.jp2 T35TMN_20180824T090549_B03_10_IN.jp2 T35TMN_20180824T090549_B04_10_IN.jp2 T35TMN_20180824T090549_B08_10_IN.jp2

rasterization (burning vectors to rasters) using gdal_rasterize

gdal_rasterize -l CLC2018_CLC2018_V2018_20 -a Code_18 -tr 10 10 -a_nodata 0 -te 918863.65 916357.41 7316593.38 5440568.3 -ot UInt16 -of GTiff -co COMPRESS=LZW -co TILED=YES -co BLOCKXSIZE=256 -co BLOCKYSIZE=256 -co NUM_THREADS=ALL_CPUS CLC2018_CLC2018_V2018_20.gdb clc_10m.tif

-tr has to be expressed taking into account the unit of the projection the data is in - in this case is EPSG:3035 LAEA which is expressed in meters

ESA-SNAP

List of SNAP modules

snap –nosplash –nogui –modules –list

Reprojection of S3 Sentinel archives using ESA GPT

Here’s a sample XML for the GPT (reprojection, band-extraction and writeback):

<graph id="Graph">
  <version>1.0</version>
  <node id="Reproject">
    <operator>Reproject</operator>
    <sources>
      <sourceProduct>${sourceProduct}</sourceProduct>
    </sources>
    <parameters>
      <crs>EPSG:4326</crs>
      <resampling>Nearest</resampling>
      <noDataValue>NaN</noDataValue>
      <includeTiePointGrids>true</includeTiePointGrids>
    </parameters>
  </node>
    <node id="BandsExtract">
    <operator>BandsExtractorOp</operator>
    <sources>
      <sourceProduct refid="Reproject"/>
      <!-- <Source>${sourceProduct}</Source> -->
    </sources>
    <parameters>
      <sourceBandNames>Oa01_radiance,Oa02_radiance,Oa03_radiance,Oa04_radiance,Oa05_radiance,Oa06_radiance,Oa07_radiance,Oa08_radiance,Oa09_radiance,Oa10_radiance,Oa11_radia\
nce,Oa12_radiance,Oa13_radiance,Oa14_radiance,Oa15_radiance,Oa16_radiance,Oa17_radiance,Oa18_radiance,Oa19_radiance,Oa20_radiance,Oa21_radiance</sourceBandNames>
      <sourceMaskNames></sourceMaskNames>
    </parameters>
  </node>
  <node id="Write_1">
    <operator>Write</operator>
    <sources>
      <sourceProduct refid="BandsExtract"/>
    </sources>
    <parameters>
      <file>${targetFile}</file>
      <!-- or, if using NetCDF: -->
      <!-- <file>${targetFile}.nc</file> -->
      <formatName>GeoTIFF-BigTIFF</formatName>
      <!-- <formatName>NetCDF-CF</formatName> -->
      <!-- <formatName>NetCDF4-BEAM</formatName> -->
    </parameters>
  </node>
</graph>

Note: NetCDF4-BEAM is NetCDF with compression!

Command execution:

sage@thor:~/alex$ 

for file in `ls /data/bid/storage0/sage-storage/homes/sage/alex/unpacked_data/`; 

do 
  /home/tardis/snap/bin/gpt reproject-bandselect.xml
  -SsourceProduct=/data/bid/storage0/sage-storage/homes/sage/alex/unpacked_data/$file/xfdumanifest.xml
  -PtargetFile="/data/bid/storage0/sage-storage/homes/sage/alex/reprojected_s3/$file";
done

Command execution when saving with BigTiff (and also using compression and tiling!)

sage@thor:~/alex$ 

for file in `ls /netcdf_archives/`;

do 
  /home/tardis/snap/bin/gpt reproject-bandselect.xml 
  -SsourceProduct=/data/bid/storage0/sage-storage/homes/sage/alex/$file/xfdumanifest.xml
  -PtargetFile="/data/bid/storage0/sage-storage/homes/sage/alex/out_dir/$file"
  -Dsnap.dataio.bigtiff.tiling.width=512
  -Dsnap.dataio.bigtiff.tiling.height=512
  -Dsnap.dataio.bigtiff.compression.type=LZW
done

Snap.parallelism

The option with GPT -Dsnap.parallelism=true enables multi-threading so it should run faster

MongoDB

Import data to a mongo database

mongoimport --username root --password pwd --authenticationDatabase admin -d database -c {col1, col2} ./path/to/file/{col1, col2}.json

or

mongorestore --username root --password pwd --authenticationDatabase admin -d database -c collection collection-data.bson

Export data from a mongo database

mongodump --username=root --password=pwd  --authenticationDatabase=admin -d database

If exporting the data like so yields in an error ending with “Unrecognized field ‘snapshot’.”, then add the –forceTableScan parameter

Create user relative to database:

use <db_name>;

db.createUser({  
 user:"username",
 pwd:"password",
 roles:[  
  {  
     role:"readWrite",
     db:"database"
  }
 ],
 mechanisms:[  
  "SCRAM-SHA-1"
 ]
})

For Mongo > 4.x you can use passwordPrompt() at pwd.

zsh

Printing a file without it’s extension

for file in `ls *.bson`; do print $file(:r); done

fzf bindings

Key Does
Ctrl + R shell command history listing
Ctrl + T Look into the filesystem for keyword
Alt + C awesome change-directory
Alexandru Munteanu
WRITTEN BY
Alexandru Munteanu
Research Assistant, PhD Student