Introduction

Let me introduce my self, Who is Fadhil? Fadhil Riyanto (real-name), is a self-taught system programmer (also backend programmer) which live in Indonesian, This is some short & dirty tutorial about computer science and litte about math.

mirrors: blog.fadhils.eu.org

Contributing

Found a mistake? You can freely DM me on telegram or just email me on

me@fadev.org (primary, fast respond)
fadhilriyanto@duck.com (gmail mirror)
fadhil.riyanto@gnuweeb.org (alternative-1)
xor@3xploit.dev (alternative-2)

Language

Because I'm indonesian, some part of this website may written in indonesian. But I will try to translating it into english language.

Note

[EN] If you found a mistake, let me know. BIG thanks!
[ID] Jika kamu menemukan sesuatu yang sepertinya itu salah, bisa langsung kasih kabar ke saya. Terimakasih.

todo

MAKE: ./server/php-fpm-config.md book
COMPLETION: ./database/postgresql_system_table
COMPLETION: ./database/postgresql_functions
MAKE: ./crypto/ca-certificates
MAKE: "freeRADIUS deploy"

System programming

How data movement changes a destination register

This snippets is from "Computer Systems A Programmer’s Perspective, by Randal E. Bryant and David R. O’Hallaron"

As described, there are two different conventions regarding whether and how data movement instruc- tions modify the upper bytes of a destination register. This distinction is illustrated by the following code sequence:

1 movabsq $0x0011223344556677, %rax %rax = 0011223344556677
2 movb $-1, %al %rax = 00112233445566FF
3 movw $-1, %ax %rax = 001122334455FFFF
4 movl $-1, %eax %rax = 00000000FFFFFFFF
5 movq $-1, %rax %rax = FFFFFFFFFFFFFFFF

In the following discussion, we use hexadecimal notation. In the example, the instruction on line 1 initializes register %rax to the pattern 0011223344556677. The remaining instructions have immediate value −1 as their source values. Recall that the hexadecimal representation of −1 is of the form FF. . .F, where the number of F’s is twice the number of bytes in the representation. The movb instruction (line 2) therefore sets the low-order byte of %rax to FF, while the movw instruction (line 3) sets the low-order 2 bytes to FFFF, with the remaining bytes unchanged. The movl instruction (line 4) sets the low-order 4 bytes to FFFFFFFF, but it also sets the high-order 4 bytes to 00000000. Finally, the movq instruction (line 5) sets the complete register to FFFFFFFFFFFFFFFF.

some useful links

https://stackoverflow.com/questions/52434073/whats-the-difference-between-the-x86-64-att-instructions-movq-and-movabsq

General embedded ASM linked in C

this is program is succesfully compiled with as, but failed to run due to SIGSEGV.

this is will fail

.data
val:
    .byte 0xF6
    .byte 0xF6
    .byte 0xF6
    .byte 0xF6
    

.globl do_asm

do_asm:
    # movl 0x12345678, %eax
    # movl 0xAABBCCDD, %edx 

    # # move edx 8 bit high into al (eax)
    # movb %dh, %al

    
    movq $9, %rax
    ret

this is will compile

.data
val:
    .byte 0xF6
    .byte 0xF6
    .byte 0xF6
    .byte 0xF6
    

.globl do_asm

.text
do_asm:
    # movl 0x12345678, %eax
    # movl 0xAABBCCDD, %edx 

    # # move edx 8 bit high into al (eax)
    # movb %dh, %al

    
    movq $9, %rax
    ret

the difference only in .text section

Move sign & zero-extends a single byte

Disini saya akan membahas apa itu instruksi MOVSBL dan MOVZBL. pertama tama silahkan baca dulu disini

https://stackoverflow.com/questions/7861095/what-does-movsbl-instruction-do

MOVSBL and MOVZBL
* MOVSBL sign-extends a single byte, and copies it into a
  double-word destination
* MOVZBL expands a single byte to 32 bits with 24 leading
  zeros, and copies it into a double-word destination

pertama tama, lihat kode asm ini.


.section .data
val:
    .byte 0xF6
    .byte 0xF6
    .byte 0xF6
    .byte 0xF6
    

.globl do_asm

.text
do_asm:
    movl $0x12345678, %eax
    movl $0xAABBCCDD, %edx 

    # move edx 8 bit high into al (eax)
    movb %dh, %al

    # reset
    movl $0x12345678, %eax
    movl $0xAABBCCDD, %edx 

    MOVSBL %dh, %eax

    # reset
    movl $0x12345678, %eax
    movl $0xAABBCCDD, %edx 

    MOVZBL %dh, %eax

    
    movq $9, %rax
    ret

dan ini hasilnya, movb biasa (tanpa hapus bagian atas data, bagian atas tetep utuh) gambar

ini hasil signed bagian atas gambar

ini hasil ketika bagian atas kita zerokan gambar

catatan movb %dh, %al memindahkan rdx bagian high ke rax bagian bawah.

movabsq

simplenya, ini kepanjangan dari move absolute quad word (64 bit) contoh kecil

itu contoh initial moveabsq int 64 bit, lalu

contoh move 1 bytes

move 16 bit, 2 bytes word

move 32 bit, 4 bytes, tapi disini keformat sebagai int32 wkwk

full 64 bit move

x86_64 register (fiks technical grade)

gambar

analisis kenapa pakai movabsq, daripada movq

movq hanya bisa accept integer

analisis patch movq pakai 0xffffffff

ada asm

kita akan memaksa dari 0x7fffffff menjadi 0xffffffff untuk membuktikan bahwa harusnya movq error, dan digantikan oleh movabsq

initial dulu

lalu, next replace address 0x555555555119 dengan

set {unsigned char[8]}0x555555555119 = {0x48, 0xc7, 0xc0, 0xff, 0xff, 0xff, 0xff, 0x90}

confirmed berubah jadi 0xffffffff disini

saatnya kita lanjutkan, apakah error

ext4 unknown RO compact features

this error come from linux kernel

	/* Check that feature set is OK for a read-write mount */
	if (ext4_has_unknown_ext4_ro_compat_features(sb)) {
		ext4_msg(sb, KERN_ERR, "couldn't mount RDWR because of "
			 "unsupported optional features (%x)",
			 (le32_to_cpu(EXT4_SB(sb)->s_es->s_feature_ro_compat) &
				~EXT4_FEATURE_RO_COMPAT_SUPP));
		return 0;
	}

which EXT4_FEATURE_RO_COMPAT_SUPP

#define EXT4_FEATURE_RO_COMPAT_SUPP	(EXT4_FEATURE_RO_COMPAT_SPARSE_SUPER| \
					 EXT4_FEATURE_RO_COMPAT_LARGE_FILE| \
					 EXT4_FEATURE_RO_COMPAT_GDT_CSUM| \
					 EXT4_FEATURE_RO_COMPAT_DIR_NLINK | \
					 EXT4_FEATURE_RO_COMPAT_EXTRA_ISIZE | \
					 EXT4_FEATURE_RO_COMPAT_BTREE_DIR |\
					 EXT4_FEATURE_RO_COMPAT_HUGE_FILE |\
					 EXT4_FEATURE_RO_COMPAT_BIGALLOC |\
					 EXT4_FEATURE_RO_COMPAT_METADATA_CSUM|\
					 EXT4_FEATURE_RO_COMPAT_QUOTA |\
					 EXT4_FEATURE_RO_COMPAT_PROJECT |\
					 EXT4_FEATURE_RO_COMPAT_VERITY |\
					 EXT4_FEATURE_RO_COMPAT_ORPHAN_PRESENT)

#define EXT4_FEATURE_RO_COMPAT_GDT_CSUM		0x0010

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/ext4/ext4.h#n2188

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/ext4/ext4.h#n2024

qemu custom cpio HDA

skrip init custom linux

qemu-system-x86_64 \-kernel ./vmlinux \
                                                 -nographic \
                                                 --append "console=tty0 console=ttyS0 panic=1 root=/dev/sda rootfstype=ext2" \
                                                 -hda root.img \
                                                 -m 1024 \
                                                 -vga none \
                                                 -display none \
                                                 -serial mon:stdio \
                                                 -no-reboot \
                                                 -initrd vfs/root.cpio.gz

generate cpio

find . | cpio -o -H newc | gzip > root.cpio.gz
``

step TCP

TCP steps

isi dulu sockaddr_storage, isi pakai inet_pton(); .so_family = AF_INET; sin_port = htons(port)
syscall socket(AF_INET, SOCK_STREAM, 0);
setsocksopt(fd, SO_REUSEADDR = 1)
bind(fd, ss_addr (cast dulu ke sockaddr_in), len nya)
listen()

penjelasan ttg kenapa pakai sockaddr_storage dipakai https://stackoverflow.com/questions/19528820/difference-between-sockaddr-and-sockaddr-storage

kenapa harus setsocksopt SO_REUSEADDR? karna jika tidak, TCP Masuk ke timewait state.

sekiranya itu aja sih step TCP ini. untuk UDP pakai aja dgram.

GDB attach

jadi ceritanya, ada bug yg kala di run di terminal, ia buggy, pas di run di gdb, dia gak error sama sekali

gw bingung dah, nah disini. ternyata ada trick nya

jalanin dulu programnya kek biasa di terminal, dah running kan
ambil pid nya pakai ps aux | grep xyz
setelah dapet pid nya, jalankan gdb dengan mode attach pid

pakai sudo gdb -p 12345

program akan freeze sejenak, pencet c, enter

nah, lakukan cara yg dipake buat reproduce bug nya, misal pakai nc buat nyepam tcp ke program. ok dapet bug nya

ni kalau kata orang internet, beberapa bug kek app multi thread agak susah di caught. pakai valgrind pun perlu flags2 khusus buat inspect sampai thread nya

hasil oprek socket programming, case socks5 server

bagian IETF SOCKS5 section 6

https://datatracker.ietf.org/doc/html/rfc1928#section-6

BND.ADDR haruslah berupa IPV4/IPV6 address, bukan domain :)

secara general, bentuk packet yg akan dikirim untuk versi IPV4 seperti ini

[0] [VER] => 0x5
[1] [REP] => 0x0
[2] [RSV] => 0x0
[3] [ATYP] => 0x1 atau 0x4
[4] [BND.ADDR] 0xac
[5] [BND.ADDR] 0x43
[6] [BND.ADDR] 0x93
[7] [BND.ADDR] 0x39
[8] [BND.PORT] 0x50 
[9] [BND.PORT] 0x50

maka perhitungannya memory awal + 4 (karna 4 field dah dipakai), start dari sini, insert 4 oktet lagi ipv4 address, lalu insert 2 oktet uint8 sebagai port.

bagian BND.PORT itu unsigned int 16 bit. jadi harus digabung bitnya + diubah dari network order ke host order

fix ncurses kernel build

suppose

*** Unable to find the ncurses libraries or the
 *** required header files.
 *** 'make menuconfig' requires the ncurses libraries.
 *** 
 *** Install ncurses (ncurses-devel) and try again.
 *** 
make[2]: *** [/home/fadhil_riyanto/linux-kernel/busybox-1.37.0/scripts/kconfig/lxdialog/Makefile:15: scripts/kconfig/lxdialog/dochecklxdialog] Error 1
make[1]: *** [/home/fadhil_riyanto/linux-kernel/busybox-1.37.0/scripts/kconfig/Makefile:14: menuconfig] Error 2
make: *** [Makefile:444: menuconfig] Error 2

maka solusi nano scripts/kconfig/lxdialog/Makefile

ubah

always         := $(hostprogs-y) dochecklxdialog

PHP-SRC variadic params

the source code: https://github.com/php/php-src/blob/ba6c00505d4218ff9fe4feb060f636524acfb125/ext/standard/array.c#L1204

mayan ngebantu jg gw ngepahami variadic params di php. ref: https://www.zend.com/resources/php-extensions/basic-php-structures

selfnote :

ZEND_PARSE_PARAMETERS_START(num_required_args, num_max_args) Z_PARAM_VARIADIC('*' or '+', destptr, argc);

catatan:

* nol atau lebih parameter
+ satu atau lebih parameter

nb: akan diupdate berkala

ref: https://wiki.php.net/rfc/fast_zpp

PHP-SRC minit()

https://github.com/php/php-src/blob/3704947696fe0ee93e025fa85621d297ac7a1e4d/main/main.c#L2009

Ketika module PHP diload ke zend engine, yang dipanggil adalah MINIT(), di step ini yg terjadi cuman memory alokasi di ZendMM, dan disini engga ada process atau thread yg dijalan, pure cuman initialisasi data

nah, yang mentrigger MINIT() adalah zend_startup_modules(), trus secara beurutan, ia jg manggil RINIT() via PHP_RINIT_FUNCTION()

struct nya zend_result (*request_startup_func)(INIT_FUNC_ARGS);

gcc compiler visibility

intresting buat diulik

class rectangle {
        int width, height;

        public:
                void set_value(int, int);
                int area();
};

void rectangle::set_value(int x, int y) 
{
        width = x;
        height = y;
}


int rectangle::area()
{
        return width * height;
}

compile pakai

g++ -Wall -c -fPIC abc.cc
g++ -o librec.so --shared abc.o
nm -C librec.so

akan ada

000000000000110e T rectangle::area()
00000000000010ea T rectangle::set_value(int, int)

nah, disitu kalau di compile pakai -fvisibility=hidden ntar pas di cek pakai nm -CD librec.so

bakalan gak ada. nah ini ada kaitannya sama __attribute__ ((visibility ("hidden")))

tambahan: https://groups.google.com/g/ode-users/c/T3qiy-4dviQ

load effective address

computes a memory address using the same arithmetic that a MOV instruction uses. But unlike the MOV instruction, the LEA instruction just stores the computed address in its target register, instead of loading the contents of that address and storing it.

Consider your first LEA instruction:

tldr nya: mov bisa mengkalkulasi operasi aritmetika dari alamat memori, trus ngambil isinya kalau lea, ia cuman mengkalkulasi alamatnya, hasil itung2an nya disimpan ke operand2

lea -0x18(%ebp), %ebx

tambahan catatan

mov (op1, op2, op3), dest

equal (pseudo): mov op1 + op2 * op3, dest

and lea op1(op2), dest

equal lea op2 + op1, dest (address of op2 + op1, store result into dest)

bahasan tentang rip https://stackoverflow.com/questions/54745872/how-do-rip-relative-variable-references-like-rip-a-in-x86-64-gas-intel-sy

AT&T asm prelude

push eax equal with: mov [esp], eax sub esp, 4

pop eax get values out of stack equal with

add rsp, 4 mov eax, [esp]

Rust stuff

Welcome in this rust stuff section.

Rust cursor, How it works

Cursor in rust basically just a in-memory implementation of Seek. see fseek() here

Imagine, we have a file. and this is representation of contents

abcde

seek start from 1, is bcde, start from 2 is cde and so-on, this is also happen when we use seek from end, for example, seek end -1 is mean e, -2 is de, etc.

From code perspective

this is signature of cursor

pub struct Cursor<T> {
    inner: T,
    pos: u64,
}

and also, this struct implement Write and Seek

so, in order to understanding this concept, let code first in C


#include <string.h>
#include <stdlib.h>
#include <stdio.h>

struct seekfd {
    FILE                *file;
};

int start_open(struct seekfd *seekfd) {
    seekfd->file = fopen("a.txt", "r");

    if (seekfd->file == NULL) {
        fprintf(stderr, "%s", "error");
        return -1;
    }
}

int do_op(struct seekfd *seekfd) {
    char allocated_buf[2048];
    memset(allocated_buf, 0, 2048);

    fseek(seekfd->file, 1, SEEK_SET);
    int ret = fread(allocated_buf, 1, 3, seekfd->file);

    printf("%s", allocated_buf);
}

int main() {
    struct seekfd seekfd;

    int ret = start_open(&seekfd);
    if (ret < 0) {
        return -9;
    }

    ret = do_op(&seekfd);
    if (ret < 0) {
        return ret;
    }
}

compile with

clang fseek.c -o p -g

next step, create dummy files named a.txt with content abcdefghijklmnopqr

first run with fseek(seekfd->file, 0, SEEK_SET); = abc second run with fseek(seekfd->file, 1, SEEK_SET); = bcd etc...

this concept is basically same in Rust

Rust context

this is small example show Seek works (in-memory) instead utilize real file

use std::io::prelude::*;
use std::io::{self, SeekFrom};
use std::io::Cursor;

fn write_bytes<W: Write + Seek>(mut w: W) -> io::Result<()> {
    w.seek(SeekFrom::Start(5))?;
    
    for i in 1..6 {
        w.write(&[i])?;
    }

    Ok(())
}


fn main() {
    let mut buf = Cursor::new(vec![0; 20]);
    write_bytes(&mut buf).unwrap();
    println!("{:?}", &buf.get_ref()); // print vector
}

this is the output

[0, 0, 0, 0, 0, 1, 2, 3, 4, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

instead, we change SeekFrom::Start(5) with SeekFrom::Start(10), the write will started at offset 10. this is the output

[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 3, 4, 5, 0, 0, 0, 0, 0]

Rust bytes has_remaining() method

this function return whatever any best left on buffer. example, let make simple tcp/ip

impl Connection {
    pub fn new(stream: TcpStream) -> Connection {
        Connection {
            stream: BufWriter::new(stream),
            buffer: BytesMut::with_capacity(4096)
        }

    }

    pub async fn read(&mut self) 
        -> std::result::Result<Option<Frame>, Box<dyn std::error::Error + Send + Sync>> {
        
        let _ = self.stream.read_buf(&mut self.buffer).await;
        println!("hasremaining: {}", self.buffer.has_remaining());
        
        while self.buffer.has_remaining() {
            let bytes = self.buffer.get_u8();
            println!("{} -> {}", bytes as char, self.buffer.has_remaining());
        }

        println!("{}", self.buffer.has_remaining());
        // Frame::check(&mut self.buffer);

        Ok(Some(Frame::Null))
    }
}

let's see the server result using ncat

ncat-result

which the input is something look like this

[22:55:41] fadhil_riyanto@integral2 /home/fadhil_riyanto [SIGINT]
> ncat 127.0.0.1 6380
abc

from this output, we know

hasremaining: true <--- this is initial, didn't mean anything
a -> true <--- a already read, but next char is b, why this is true
b -> true <--- next is c, also true
c -> true <--- next is \n char, also true

 -> false <--- this is false because no next chars
false <--- confirm

Also, get_u8 returns the first byte in the buffer. Each time it's called, it advances the position, so subsequent calls return the next bytes.

How Rust into() Box + Error analysis

suppose, we have function return signature like this

std::result::Result<Option<Frame>, Box<dyn std::error::Error + Send + Sync>>

and then, we return something like this from our function

return Err("connection reset".into())

how the into() works?

analysis

look at this snippet

#[cfg(not(no_global_oom_handling))]
#[stable(feature = "rust1", since = "1.0.0")]
impl<'a> From<&str> for Box<dyn Error + Send + Sync + 'a> {
    /// Converts a [`str`] into a box of dyn [`Error`] + [`Send`] + [`Sync`].
    ///
    /// [`str`]: prim@str
    ///
    /// # Examples
    ///
    /// ```
    /// use std::error::Error;
    /// use std::mem;
    ///
    /// let a_str_error = "a str error";
    /// let a_boxed_error = Box::<dyn Error + Send + Sync>::from(a_str_error);
    /// assert!(
    ///     mem::size_of::<Box<dyn Error + Send + Sync>>() == mem::size_of_val(&a_boxed_error))
    /// ```
    #[inline]
    fn from(err: &str) -> Box<dyn Error + Send + Sync + 'a> {
        From::from(String::from(err))
    }
}

if we go deeper into From::from, we found this

#[inline]
    fn from(err: String) -> Box<dyn Error + Send + Sync + 'a> {
        struct StringError(String);

        impl Error for StringError {
            #[allow(deprecated)]
            fn description(&self) -> &str {
                &self.0
            }
        }

        impl fmt::Display for StringError {
            fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
                fmt::Display::fmt(&self.0, f)
            }
        }

        // Purposefully skip printing "StringError(..)"
        impl fmt::Debug for StringError {
            fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
                fmt::Debug::fmt(&self.0, f)
            }
        }

        Box::new(StringError(err))
    }

nb: actually StringError is string aliases. then, we see in this section. we created a heap-allocated using Box::new(StringError(err)).

Rust buffer advance

this simple article show how buffer advance works

use bytes::Buf;

fn main() {
    let mut buf = &b"hello world"[..];

    println!("{}", String::from_utf8_lossy(buf.chunk()));

    buf.advance(6);
    println!("{}", String::from_utf8_lossy(buf.chunk()));
}

and the result

hello world
world

Rust RESP get_line analysis

if we look at simple tokio redis example (mini redis), we can see how the data is parsed. look at this

the resp is generally read something like this `+abc\r\n'

this is the code that actually crop down the buffer

fn get_line<'a>(src: &mut Cursor<&'a [u8]>) -> Result<&'a [u8], Error> {
    // Scan the bytes directly
    let start = src.position() as usize;
    // Scan to the second to last byte
    let end = src.get_ref().len() - 1;

    for i in start..end {
        if src.get_ref()[i] == b'\r' && src.get_ref()[i + 1] == b'\n' {
            // We found a line, update the position to be *after* the \n
            src.set_position((i + 2) as u64);

            // Return the line
            return Ok(&src.get_ref()[start..i]);
        }
    }

    Err(Error::Incomplete)
}

and I will show you how they works (algorithm)

variables setup

we have start and end, which consist of

let start = src.position() as usize;
let end = src.get_ref().len() - 1;

the src.position() always return 0, also depending on the context. src.get_ref().len() return the length of the buffer, for example, +rand\n\n has length 7.

because we wan't to read two chars at the end, which is \r and \n

so we substract length with 1, this is avoid buffer overflow

suppose this situation: buf: +rand\n\n

when we add everything with 1, we got this idx: 012345 6 7, it is idx[7]is found? no, so we need to sub with 1, the idx is something like this idx: 012345 6, no overflow

then we check the i and i + 1, make sure there is \r\n

set position

now, we set the cursor to the next incoming buffer. which after \r\n

src.set_position((i + 2) as u64);

suppose:

+rand\r\n+randomsomething\r\n
012345 6 ^ 
^^   ^   this is i + 2, indicates next buffer
ii+1 orig i

postgresql - get() generic function signature

In someday, i found

#[track_caller]
    pub fn get<'a, I, T>(&'a self, idx: I) -> T
    where
        I: RowIndex + fmt::Display,
        T: FromSql<'a>,
    {
        match self.get_inner(&idx) {
            Ok(ok) => ok,
            Err(err) => panic!("error retrieving column {}: {}", idx, err),
        }
    }

some sort of notes:

https://doc.rust-lang.org/rust-by-example/generics.html
https://serokell.io/blog/rust-generics

how to call it

let data2 = data_actual[0].get::<_, i32>(0);

Rust build logging

rust build usually don't show any output if you try to println!() in build.rs, in order to force this, use

println!("cargo:warning=start build!");

result

warning: ourcrate-rs@0.1.0: Something went wrong!

all docs: https://doc.rust-lang.org/cargo/reference/build-scripts.html

Rust tonic gRPC server build process

topik ini khusus membahas gRPC di rust,

build

contoh template build rs:

use std::{env, path::PathBuf};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // let out_dir = PathBuf::from(env::var("OUT_DIR").unwrap());
    let out_dir = PathBuf::from(
        "/home/fadhil_riyanto/BALI64/fadev-service-rs/rpc/src/pb".to_string()
    );

    println!("cargo:warning=start build!");
    
    tonic_build::configure()
        .build_server(true)
        .build_client(false)
        .file_descriptor_set_path(out_dir.join("random_service.bin"))
        .compile_protos(
            &["proto/random_service.proto"], 
            &["proto"]
        )?;


    Ok(())
}

nb:

file_descriptor_set_path: dipakai sebagai reflection, bakal ketemu kalau pakai grpcurl.
build_server dan build_client: disini ada 2 mode build, untuk server atau untuk client. maka kita set server true, client false
compile_protos: masing2 berisi 2 param yg menerima slice, param pertama berisi lokasi protobuf, param kedua berisi root foldernya.

anda bisa lihat-lihat contoh lain disini: https://github.com/hyperium/tonic/tree/master/examples

optional:

use .out_dir() ketimbang harus nurut dengan apa yang OUT_DIR katakan

OUT_DIR

what it is, simplenya OUT_DIR ini akan diset jika protobuf sudah di compile, dalam artian, file *.proto diubah jadi kumpulan file *.rs yang bakal di include sama macro, kemudian di compile

urutan compile adalah, macro dievaluasi dulu, baru compile, sampai sini masuk akal. ini macro signature nya

#[macro_export]
macro_rules! include_proto {
    ($package: tt) => {
        include!(concat!(env!("OUT_DIR"), concat!("/", $package, ".rs")));
    };
}

tambahan info bisa baca2 sendiri disini: https://doc.rust-lang.org/cargo/reference/environment-variables.html#environment-variables-cargo-sets-for-build-scripts, dan jg biasanya ini akan diinclude ke sebuah file handler. kira kira seperti itu

Rust tokio-postgres connection config example

first, this is example from the documentation

 let (client, conn_ctx) = tokio_postgres::connect("host=localhost user=postgres", NoTls);

then, we want to convert it into config like ctx. see this

use tokio_postgres::{NoTls, Error};

#[tokio::main]
async fn main() {
        let mut config = postgres::config::Config::new();
        config.host("localhost");
        config.user("fadhil_riyanto");
        config.password("1921684338");

        let client_ret = config.connect(NoTls);


}

Math

Welcome in this section.

Set operations formula

Union of Two Sets (cardinality)

\[ |A \cup B| = |A| + |B| - |A \cap B| \]

how to read: whole A and B add together, its make a data duplication on center of venn diagram, we remove that duplication using $ |A \cap B| $

Intersection (steps)

its flip of Union \[ |A \cup B| = |A| + |B| - |A \cap B| \] \[ |A \cap B| = |A| + |B| - |A \cup B| \]

Intersection of three sets

\[ |A \cup B \cup C| = |A| + |B| + |C| - |A \cap B| - |A \cap C| - |B \cap C| + |A \cap B \cap C| \]

This is proof of concept by nice guy on internet: https://www.youtube.com/watch?v=vVZwe3TCJT8.

Difference

\[ |A - B| = |A| - |A \cup B| \]

Symetric difference

1. General formula

\[ A \space \triangle \space B = (A \cup B) - (A \cap B) \] Cardinals version: \[ | A \space \triangle \space B | = |A - B| + |B - A| \] \[ | A \space \triangle \space B | = |A| + |B| - 2|A \cap B| \]

2. Symetric difference properties

$ A \space \triangle \space B = B \space \triangle \space A $
$ (A \space \triangle \space B) \space \triangle C = A \space \triangle \space (B \space \triangle C) $
$ (A \space \triangle \space \emptyset) = A $, why?
- $A \space \triangle \space B = (A \cup B) - (B \cap A) $
- $A \space \triangle \space \emptyset = (A \cup \emptyset) - (\emptyset \cap A) $, but
- $ (A \cup \emptyset) = A $ and $ (\emptyset \cap A) = \emptyset $
- $ (A - \emptyset) = A $
$ (A \space \triangle \space A) = \emptyset $

Set stuff

there is some intresting question about discrete math

why $ \mathcal{P}(\emptyset) = \lbrace \emptyset \rbrace $

consider $ \mathcal(S) $ where $ S $ is a set, for example

if $ S = \lbrace a \rbrace $, then $ \mathcal{P}(S) = \lbrace \emptyset, \lbrace a \rbrace \rbrace $, same as $ \mathcal{P}(a) $
if $ S = \lbrace a, b \rbrace $, then $ \mathcal{P}(S) = \lbrace \emptyset, \lbrace a \rbrace, \lbrace b \rbrace, \lbrace a, b \rbrace \rbrace $, same as $ \mathcal{P}(a, b) $
if $ S = \lbrace \emptyset \rbrace $, then $ \mathcal{P}(S) = \lbrace \emptyset, \lbrace \emptyset \rbrace \rbrace $, same as $ \mathcal{P}(\emptyset) $

so, because $ \lbrace \rbrace $ usually hidden, $ \mathcal{P}(\emptyset) = \lbrace \emptyset \rbrace $

let $ A = \emptyset, B = \emptyset $, how to $ A \times B $

Consider $ A \times B = \lbrace (a,b) | a \in A, b \in B \rbrace $

because

A is empty, no elements can be paired
B is also empty

conlusion: no paired element = empty

proof by cardinality $ |A| = 0, |B| = 0, \text{so, when } |A \times B| = |A| \times |B| = 0 \times 0 = 0 $

does $ P(\emptyset) = \lbrace \lbrace \rbrace \rbrace = \lbrace \emptyset \rbrace $ ?

true, this is due $ \lbrace \emptyset \rbrace $ same as $ \lbrace \lbrace \rbrace \rbrace $, then consider example $ P(a) = \lbrace \lbrace \rbrace, \lbrace a \rbrace\rbrace $

Suppose X and Z are independent of each other

$ | \mathcal{P}(X \cap Z)| $
$ | X - Z | $
$ | X \oplus Z | $
$ | X \cap Z | $
$ | \mathcal{P}(X) \cup \mathcal{P}(Z)| $
$ | \overline{\mathcal{P}(X) \cup \mathcal{P}(Z)}| $

nb: bagian ini belum jadi

Logika

Ringkasan ini dibuat sebagian sebagian dari Kuliah IF1220 Matematika Diskrit ITB oleh bapak Rinaldi Munir. dan berbagai sumber seperti Wikipedia, ChatGPT dsb

Proposisi

adalah pernyataan bahwa suatu dua perbandingan adalah sama besar, contoh

$ \frac{a}{b}=\frac{c}{d} $
$ 4 = 2 * 2 $
13 adalah bilangan ganjil
Untuk sembarang bilangan bulat $ n >= 0 $, maka $ 2n $ adalah genap

contoh yang bukan

Isilah gelas tersebut dengan air
$ x + 3 = 8 $
$ x > 3 $

dan, Proposisi dilambangkan dengan huruf kecil seperti $ p, q, r, ... $

Bentuk bentuk nya

Proposisi atomik bentuk proposisi tunggal, contoh
- 2n selalu genap untuk n=0, 1, 2, …
- Ibukota Maluku Utara adalah Ternate
proposisi majemuk diantaranya:
- conjunction (and): $ \wedge $
- disjunction (or): $ \vee $
- negation (!n): $ ~ $
- exclusive disjunction (xor): $ \oplus $

Table

p	q	r	p ∧ q	¬q	¬q ∧ r	(p ∧ q) ∨ (¬q ∧ r)
T	T	T	T	F	F	T
T	T	F	T	F	F	T
T	F	T	F	T	T	T
T	F	F	F	T	F	F
F	T	T	F	F	F	F
F	T	F	F	F	F	F
F	F	T	F	T	T	T
F	F	F	F	T	F	F

nb:

Tautologi adalah pernyataan logika yang selalu benar, tidak peduli nilai kebenaran dari komponen-komponennya. contoh

p	q	p ∧ q	¬(p ∧ q)	p ∨ ¬(p ∧ q)
T	T	T	F	T
T	F	F	T	T
F	T	F	T	T
F	F	F	T	T

Kontradiksi adalah pernyataan logika yang selalu salah, tidak peduli nilai kebenaran dari komponen-komponennya. contoh

p	q	p ∧ q	p ∨ q	¬(p ∨ q)	(p ∧ q) ∧ ¬(p ∨ q)
T	T	T	T	F	F
T	F	F	T	F	F
F	T	F	T	F	F
F	F	F	F	T	F

- ekivalen Intinya, jika dioperasikan, dia punya hasil tabel kebenaran yang identik, walau rumusnya beda.

gambar wikipedia hukum de morgan

p	q	p ∧ q	¬(p ∧ q)	¬p	¬q	¬p ∨ ¬q
T	T	T	F	F	F	F
T	F	F	T	F	T	T
F	T	F	T	T	F	T
F	F	F	T	T	T	T

Hukum hukum nya

Bahasa Indonesia	Bahasa Inggris	Contoh
Hukum Identitas	Identity Law	`p ∧ T ≡ p`, `p ∨ F ≡ p`
Hukum Dominasi	Domination Law	`p ∨ T ≡ T`, `p ∧ F ≡ F`
Hukum Idempotensi	Idempotent Law	`p ∨ p ≡ p`, `p ∧ p ≡ p`
Hukum Negasi	Negation Law	`p ∨ ¬p ≡ T`, `p ∧ ¬p ≡ F`
Hukum Komutatif	Commutative Law	`p ∨ q ≡ q ∨ p`, `p ∧ q ≡ q ∧ p`
Hukum Asosiatif	Associative Law	`(p ∨ q) ∨ r ≡ p ∨ (q ∨ r)`
Hukum Distributif	Distributive Law	`p ∨ (q ∧ r) ≡ (p ∨ q) ∧ (p ∨ r)`
Hukum De Morgan	De Morgan’s Laws	`¬(p ∧ q) ≡ ¬p ∨ ¬q`, `¬(p ∨ q) ≡ ¬p ∧ ¬q`
Hukum Involusi	Double Negation / Involution	`¬(¬p) ≡ p`
Hukum Implikasi	Implication Law	`p → q ≡ ¬p ∨ q`
Hukum Biimplikasi	Biconditional Law	`p ↔ q ≡ (p → q) ∧ (q → p)`

Implikasi

Disebut juga proposisi bersyarat, seperti jika x maka y, notasinya $ p \rightarrow q$. $ p $ nya adalah condition, $ q $ nya adalah conlusion

p	q	p → q
T	T	T
T	F	F
F	T	T
F	F	T

versi versinya jika dijadikan teks

Jika p, maka q (if p, then q)
Jika p, q (if p, q)
p mengakibatkan q (p implies q)
q jika p (q if p)
p hanya jika q (p only if q)
p syarat cukup untuk q (p is sufficient condition for q)
q syarat perlu bagi p (q is necessary condition for q)
q bilamana p (q whenever p)
q mengikuti dari p (q follows from p)

penjelasan kenapa $ F \rightarrow F = T $

“If I win the lottery, I’ll buy you a car.”
I didn't win the lottery → (False)

I didn't buy a car → (False)

dan juga $ P \rightarrow Q $ sebenarnya sama dengan $ \sim{P} \vee Q $

Penjelasan kenapa $ \sim (p \rightarrow q )$ itu sama dengan $ p \space \wedge \sim{q} $

Kita tahu bahwa $ p \rightarrow q $ itu sebenarnya sama dengan $ \sim{p} \vee p $ , maka steps yang dibutuhkan hanya

$ \sim(p \rightarrow q) $
$ \sim(\sim{p} \vee q) $
$ p \space \wedge \sim{q} $

tabel varian implikasi (Proporsi bersyarat)

Konvers (kebalikan): $ q \rightarrow p $
Invers : $ \sim p \rightarrow \sim q $
Kontraposisi : $ \sim q \rightarrow \sim p $

p	q	¬p	¬q	p → q	q → p	¬p → ¬q	¬q → ¬p
T	T	F	F	T	T	T	T
T	F	F	T	F	T	T	F
F	T	T	F	T	F	F	T
F	F	T	T	T	T	T	T

Bi-impication

Intinya, operand kanan kiri harus sama, entah sama sama true, atau sama sama false. notasinya: $ p \leftrightarrow q $

tabel kebenaran

p	q	p ↔ q
T	T	T
T	F	F
F	T	F
F	F	T

contoh

p	q	p ↔ q	p → q	q → p	(p → q) ∧ (q → p)
T	T	T	T	T	T
T	F	F	F	T	F
F	T	F	T	F	F
F	F	T	T	T	T

analogi simple:

Jika suatu bilangan genap, maka habis dibagi 2: $ \text{true} \leftrightarrow \text{true} = \text{true} $
Jika suatu bilangan bukan genap, maka tidak akan habis dibagi 2: $ \text{false} \leftrightarrow \text{false} = \text{true} $

kumpulan soal Matematika Diskrit

No. 1

Dalam sebuah survei terhadap 120 orang, diperoleh data sebagai berikut:

65 orang suka kopi
70 orang suka teh
50 orang suka cokelat
30 orang suka kopi dan teh
28 orang suka kopi dan cokelat
26 orang suka teh dan cokelat
15 orang suka semua minuman
Semua orang minimal menyukai satu jenis minuman

Tentukan:

Banyak orang yang hanya suka kopi
Banyak orang yang suka teh dan cokelat, tapi tidak suka kopi
Banyak orang yang suka tepat dua jenis minuman
Banyak orang yang hanya suka satu jenis minuman

No. 2

Jika diketahui

S = {1,2,3 …, 15},
A = { 2,3,5,7},
B = { 1,3,5,7,9},
C = {2,4,6,8}

Tentukan :

A ∪ B ∪ C
A ∩ C
B - A
(A’ ∩ B) - C
(B - C) ∩ A’

No. 3

Dalam survei terhadap 80 mahasiswa, diperoleh data:

45 mahasiswa suka Matematika
38 mahasiswa suka Fisika
40 mahasiswa suka Kimia
20 mahasiswa suka Matematika dan Fisika
25 mahasiswa suka Fisika dan Kimia
18 mahasiswa suka Matematika dan Kimia
10 mahasiswa suka ketiga mata kuliah
8 mahasiswa tidak suka semua mata kuliah

tentukan

Banyak mahasiswa yang suka hanya Matematika
Banyak mahasiswa yang suka tepat dua mata kuliah
Banyak mahasiswa yang suka Matematika atau Kimia tapi tidak Fisika
Banyak mahasiswa yang suka hanya satu mata kuliah

Catatan

format latex menyusul

asal usul rumus sudut vektor

awal mula dari rumus entah berantah ini. link untuk oprek rumus: https://proofwiki.org/wiki/Cosine_Formula_for_Dot_Product

\[ \vec{A} \cdot \vec{B} = |\vec{A}| |\vec{B}| cos(\theta) \]

lalu \[ \frac{\vec{A} \cdot \vec{B}}{|\vec{A}| |\vec{B}|} = cos(\theta) \]

lalu kita bisa balik, menjadi \[ cos(\theta) = \frac{\vec{A} \cdot \vec{B}}{|\vec{A}| |\vec{B}|} \]

\[ arccos(\frac{\vec{A} \cdot \vec{B}}{|\vec{A}| |\vec{B}|}) = \theta \]

solved!

transformasi elementer

beberapa notasi aneh disini diderive dari PPT kampus.

$ H_{ij}A $: Swap baris i dengan baris j
$ H_i^{(k)} $: mengalikan baris i dengan konstanta k
$ H_{ij}^{(k)} $: hitung dulu konstanta k dikali baris j, lalu hasilnya ditambah baris i, simpan jg hasilnya ke baris i
$ H_{i \space \space \space \space \space \space \space j}^{\space \space (k) \space \space \space (z)}(A)_i $: intinya, baris ke i dikalikan dgn k, kalau udah lanjut hitung baris j dikalikan z. setelah itu hasilnya tentusaja simpan ke i hasil itung2 an tadi.

Cryptography

Alur konek ke RPC server telegram

step ini dibuat oleh seseorang dari masa lalu sekitar tahun 2020 yang akunnya telah terhapus, credit kepada Butthx

`originals written in telegram chat, but because I want to preserve it. I write it here

step 1:

Bikin koneksi ke server telegram pakai TCP yang tersedia (disini gwa pakai TCPIntermediate)

step 2:

Bikin random nonce dengan value int128

step 3:

Kirim bytes dari ReqPqMulti dengan value bytes dari nonce tadi.

step 4:

pick salah satu dari beberapa publick fingerprints yang diberikan dari results step3.

step 5:

Lakuin faktor bilangan pq yang diberikan dari results step3, dapetin lagi hasil pembagian dari pq tadi dengan hasil faktor, terus urutkan dari yang terkecil. untuk yang kecil di kasih nama p dan yang besar kasih nama g.

step 6:

bikin nonce baru (jangan ubah nonce yang lama) dengan value random int256.

step 7:

bikin hash (sha1) dari bytes PQInnerData ke telegram: bytes[pq,bytes p,bytes q,nonce,nonce baru,server nonce (dapet dari step3)]

step 8:

bikin padding dengan random bytes dengan panjang -(length bytes PQInnerData + length sha1) mod 255

step 9:

encrypt bytes[sha,bytes PQInnerData,padding] dengan RSA (pakai publik fingerprints tadi)

step 10:

kirim bytes ReqDhParams: bytes[nonce,server nonce, bytes p, bytes q, encrypted data (step9) ,fingerprint]

step 11:

bikin temporary aes key: bytes[sha1[bytes nonce baru, bytes server nonce],sha1[bytes server nonce, bytes nonce baru] slice 0 – 12]

step 12:

bikin temporary aes initial vector (IV): bytes[sha1[bytes server nonce, bytes nonce baru] slice 12 – end, sha1[bytes nonce baru,bytes nonce baru], bytes[nonce baru] slice 0 – 4 ]

step 13:

decrypt encryptedAnswer (dapet dari step 10) menggunakan AES-256-IGE pakai key sama iv tadi.

step 14:

parse decrypted answer tadi ke TLSchema/TLObject

step 15:

bikin random int(panjang 256 bytes, bukan int256 ygy) kita beri nama b

step 16:

dapetin g_b dengan (g ** b) mod dh_prime

g, dh_prime → dapet dari step 14

step 17:

bikin hash sha1 dari bytes[ClientDhInnerData[nonce, server nonce, 0,unsigned big bytes g_b length 256]]

step 18:

bikin padding dengan length -(length bytes ClientDhInnerData + length sha1) mod 16

step 19:

encrypt bytes[bytes step 17,bytes ClientDhInnerData, padding ] dengan AES-256-IGE, key sama iv masih menggunakan yang step 11.

step 20:

kirim bytes[SetClientDhParams[nonce, server nonce, bytes step 19]]

step 21:

Bikin auth key dengan big-unsigned-bytes[big-u-int(g_a) ** b mod dhPrime] length 256

step 22:

Lakuin security check..

notes: Untuk format pengiriman bytes:

bytes[sessionId,messageId,length content,content]
karena belum ada authkey, jadi sessionId  = Buffer.alloc(8)

untuk dapetin real data dari telegram, slice bytes yang di terima dari 20 – end

pengiriman content selalu bentuk bytes, jadi semua parameter jadiin bytes :)

/**
 * tgsnake - Telegram MTProto framework for nodejs.
 * Copyright (C) 2022 butthx <https://github.com/butthx>
 *
 * THIS FILE IS PART OF TGSNAKE
 *
 * tgsnake is a free software : you can redistribute it and/or modify
 * it under the terms of the MIT License as published.
 */

export const DCTest = {
  1: '149.154.175.10',
  2: '149.154.167.40',
  3: '149.154.175.117',
};
export const DCProd = {
  1: '149.154.175.53',
  2: '149.154.167.51',
  3: '149.154.175.100',
  4: '149.154.167.91',
  5: '91.108.56.130',
};
export const DCProdMedia = {
  2: '149.154.167.151',
  4: '149.154.164.250',
};
export const DCTestIPV6 = {
  1: '2001:b28:f23d:f001::e',
  2: '2001:67c:4e8:f002::e',
  3: '2001:b28:f23d:f003::e',
};
export const DCProdIPV6 = {
  1: '2001:b28:f23d:f001::a',
  2: '2001:67c:4e8:f002::a',
  3: '2001:b28:f23d:f003::a',
  4: '2001:67c:4e8:f004::a',
  5: '2001:b28:f23f:f005::a',
};
export const DCProdMediaIPV6 = {
  2: '2001:067c:04e8:f002:0000:0000:0000:000b',
  4: '2001:067c:04e8:f004:0000:0000:0000:000b',
};
export function DataCenter(
  dcId: number,
  testMode: boolean,
  ipv6: boolean,
  media: boolean
): [ip: string, port: number] {
  if (testMode) {
    return [ipv6 ? DCTestIPV6[dcId] : DCTest[dcId], 80];
  } else {
    if (media) {
      return [ipv6 ? DCProdMediaIPV6[dcId] : DCProdMedia[dcId], 443];
    } else {
      return [ipv6 ? DCProdIPV6[dcId] : DCProd[dcId], 443];
    }
  }
}

ca-certificates

draft: mengoprek kriptografi dibalik HTTPS & konsep public private key pair dan juga peran CA authority, kenapa mereka harus dimasukkan ke package ???

dan juga kenapa MITM (Man in the Middle) bisa melogging jaringan, ketika certificate telah dipasang. kenapa hanya dengan certificate mitm bisa mendecode data yang lewat.

Server

Cgit setup

Ku mau share pengalaman bikin self hosting git server, pakai cgit, biar engga kalah aja sama git.kernel.org dan kernel.dk wkwk

step satu, include dir nya sites-available spt biasa, contoh config nya https://gist.github.com/fadhil-riyanto/328f2cc0607eb58b713cdeaa473d08ee, lalu config untuk sites-available nya https://gist.github.com/fadhil-riyanto/c2e26de30c97600e4ee9342aefbd6ece

nb: install fcgiwrap dulu

lalu edit /etc/cgitrc, contoh config: https://gist.github.com/fadhil-riyanto/4033e6ce59a35568c31dbd5e5a1d29bd

harusnya sampai sini, cgit dah bisa diakses, cuman empty repository aja.

ok, lalu kita ke permission2 nya, pertama2 pakai akun root, tldr

cd /root
mkdir git
cd git
mkdir user1, misal aja mkdir fadhil_riyanto
cd fadhil_riyanto

clone git apapun disini, misal git clone https://github.com/fadhil-riyanto/reponame.git --bare (harus pakai bare)

posisi dir ini di /root/git/fadhil-riyanto/reponame.git

ok, balik ke parent dir, posisi di /root/git lakukan chown -R fadhil_riyanto fadhil-riyanto, jika engga nanti error "git submodule update" failed with 'fatal: detected dubious ownership in repository at...' etc...

ok, saatnya push git remote add cgit fadhil_riyanto@123.123.111.222:/root/git/fadhil-riyanto/reponame.git

config HTTPS nginx

Cara config HTTPS di server pakai certbot (NGINX)

apt install certbot python3-certbot-nginx
verify config na pakai nginx -t
generate cert na pakai certbot --nginx -d namasite.com

ganti namasite.com dengan embel2 di /etc/nginx/sites-enabled/somefile.conf

didalamnya kan ada server_name, nah diganti dgn itu

Setup virtual network interface

cara setup virtual ethernet + assign ip nya

sudo ip link add veth0 type dummy

bisa dicek pakai ip addr

[11:11:35] fadhil_riyanto@integral2 /etc/systemd/network  
> ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp2s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 2c:4d:54:c6:f2:0c brd ff:ff:ff:ff:ff:ff
    altname enx2c4d54c6f20c
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f0:03:8c:66:9c:21 brd ff:ff:ff:ff:ff:ff
    inet 192.168.5.44/24 brd 192.168.5.255 scope global dynamic noprefixroute wlan0
       valid_lft 581sec preferred_lft 581sec
    inet 192.168.5.197/24 metric 1024 brd 192.168.5.255 scope global secondary dynamic wlan0
       valid_lft 582sec preferred_lft 582sec
    inet6 fe80::5574:5771:be47:2f7b/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::f203:8cff:fe66:9c21/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
4: veth0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 76:4d:3f:3d:11:7f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::744d:3fff:fe3d:117f/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

lalu assign ip nya

sudo ip addr add 192.168.100.10/24 dev veth0
sudo ip link set veth0 up

dan bisa diping juga

untuk remove interface nya bisa pakai sudo ip link delete vnet0

some sorts of equalivalent command

Create bridge: brctl addbr br0 ip link add br0 type bridge
Delete bridge: brctl delbr br0 ip link delete br0
Add interface to bridge: brctl addif br0 eth0 ip link set eth0 master br0
Remove interface from bridge: brctl delif br0 eth0 ip link set eth0 nomaster
Show bridge info: brctl show or brctl show br0 bridge link show or ip link show master br0
Enable/disable STP: brctl stp br0 on/off bridge link set dev eth0 guard off/on (note: STP enable/disable logic is reversed)

ip aliasing

satu interface bisa punya lebih dari 1 IP dinamai sebagai IP aliasing. contoh

eth0 bisa punya 2 ip dengan cara

ip addr add 192.168.1.100/24 dev eth0
ip addr add 192.168.1.101/24 dev eth0

atau bisa juga ada 2 interface, katakanlah eth0 dengan eth1, masing2 punya ip nya sendiri2 dengan cara

ip addr add 192.168.1.10/24 dev eth0
ip addr add 10.0.0.5/24 dev eth1

tambahan

untuk ngeremove salah satu ip nya, bisa dengan ip addr del <IP NYA>/<CIDR> dev <IFACE>

Setup bridge enp0s16u1

setup dulu bridgenya pakai ip link add br0 type bridge
buat enp0s16u1 sebagai master dari br0, pakai ip link set enp0s16u1 master br0
- catatan, wlan0 tidak bisa di add. https://bbs.archlinux.org/viewtopic.php?id=264651, https://wiki.archlinux.org/title/Network_bridge#Wireless_interface_on_a_bridge
lalu enable semua interface nya
- sudo ip link set dev enp0s16u1 up
- sudo ip link set dev br0 up
set dhcp sudo dhclient br0
- nb, ditahap ini, harusnya sudah dapat ip dan bisa ping keluar.
hapus ip bawaan enp0s16u1 dan pakai br0 sebagai default interface pakai sudo ip addr flush dev enp0s16u1

bisa di test dengan

hapus dahulu ip yang tadi diberikan ke br0 pakai sudo dhclient -r br0, pastikan ip nya kosong
lalu hapus ip milik enp0s16u1
test ping
lalu lakukan lagi sudo dhclient br0

reverse command

Pakai ini untuk in-case ada kesalahan

hapus ip br0

ip addr show br0
sudo ip addr del 192.168.1.10/24 dev br0

tambahan, kadang tidak bener2 kehapus karna mungkin NetworkManager, systemd-networkd, dhclient aktif.

nb: untuk dhcp kadang NetworkManager sama systemd-networkd bertabrakan, solusinya stop dahulu pakai

sudo systemctl stop systemd-networkd.socket
sudo systemctl stop systemd-networkd

next, buat bridge ke VM. topologi

   [Internet]
       |
     Router
       |
     eth0 (on host) → br0 (bridge)
                      |
    +--------+--------+--------+
    |        |        |        |
  VM1      VM2      VM3      etc.
  ↳ 192.168.12.101   192.168.12.102   ...

mulai dari sini akan utak atik veth

setup veth manual

sudo ip link add veth0 type veth peer name veth0-peer
sudo ip link set veth0 master br0
sudo ip link set veth0 up
sudo ip link set veth0-peer up
sudo dhclient veth0-peer

topologinya kira2 seperti ini

                +---------------------------+
                |        Linux Host         |
                |                           |
                |   +-------------------+   |
                |   |       br0         |   |  ← bridge acting as a virtual switch
                |   +--------+----------+   |
                |            |              |
                |         veth0            ... ← more veths for other containers/VMs
                |            |
                +------------+--------------+
                             |
                             | (virtual cable)
                             |
                         veth0-peer
                             |
                    +------------------+
                    |   Container/VM   |  ← gets its own IP
                    +------------------+

lanjut assign veth-peer ke qemu (pakai tap device)

topology:

            Linux Host
       +------------------+
       |      br0         |  ← acts as switch
       +--+-----------+---+
          |           |
        eth0         tap0  ← tap device for QEMU VM
                        |
                   QEMU VM

attach

sudo ip tuntap add dev tap0 mode tap
sudo ip link set tap0 up
sudo ip link set tap0 master br0

qemu

sudo qemu-system-x86_64 \
  -enable-kvm \
  -m 2048 \
  -hda ubuntu.qcow2 \
  -netdev tap,id=net0,ifname=tap0,script=no,downscript=no \
  -device virtio-net-pci,netdev=net0

test qemu mikrotik

qemu-system-x86_64 \
                -enable-kvm \
                -boot order=d \
                -drive file=chr.qcow2,format=qcow2 \
                -m 4G \
                -smp 4 \
                -net user,hostfwd=tcp::10022-:22,hostfwd=tcp::8080-:8080 \
                -netdev tap,id=net0,ifname=tap0,script=no,downscript=no \
                -device virtio-net-pci,netdev=net0

untuk test nya, bisa dengan

cek ip mikrotik pakai /ip address print
lalu ip netns exec namespace_satu ping <IP NYA>
Bisa juga pakai dummy http server php, caranya sudo ip netns exec namespace_satu php -S 192.168.12.82:1234 (cek dahulu ip namespace pakai ip addr)

contoh setup pakai namespace

create dulu pakai sudo ip netns add namespace_satu
set sudo ip link set veth0-peer netns namespace_satu
cek dahulu, pastikan didalamnnya masih kosong ip nya, pakai sudo ip netns exec namespace_satu ip addr
lalu kita nyalakan veth0-peer nya sudo ip netns exec namespace_satu ip link set dev veth0-peer up, tapi tetap saja masih belum dapat ip. maka next kita exec dhclient pakai sudo ip netns exec namespace_satu dhclient veth0-peer
test ping keluar

Setup shadowsocks proxy server

shadowsocks-libev is available in the official repository for Debian 9("Stretch"), unstable, Ubuntu 16.10 and later derivatives:

sudo apt update
sudo apt install shadowsocks-libev

then, create a file config on /etc/shadowsocks-libev/config.json which has contents something like this

{
    "server":["::1", "10.1.1.4"],
    "mode":"tcp_and_udp",
    "server_port":8388,
    "local_port":1080,
    "password":"1u21wW3E0bwu",
    "timeout":86400,
    "method":"chacha20-ietf-poly1305"
}

in this case, I use 10.1.1.4 as local ip, then forwarded through firewall to the public addr.

then, sudo systemctl start shadowsocks-libev

ref: https://shadowsocks.org/doc/configs.html

connect

Alpine htop install on live QEMU

create disk first

qemu-img create -f qcow2 alpine-1.qcow2 1G

start qemu

download iso here https://dl-cdn.alpinelinux.org/alpine/v3.21/releases/x86_64/alpine-virt-3.21.3-x86_64.iso

qemu-system-x86_64 \
      -enable-kvm \
      -m 256 \
      -smp 4 \
      -cpu host \
      -drive file=pc-1.qcow2,format=qcow2 \
      -cdrom alpine-virt-3.21.3-x86_64.iso \
      -boot d \
      -net nic \
      -net user -nographic

default username is root, password left blank, just enter

set eth0 up & DHCP

ip link set dev eth0 up
udhcpc -i eth0
ping 1.1.1.1 (testing)

for /etc/resolv.conf, change to nameserver 1.1.1.1

patch apk configuration

cp /etc/apk/repositories /etc/apk/repositories-old

--- repositories-old
+++ repositories
@@ -1 +1,2 @@
-/media/cdrom/apks
+http://dl-cdn.alpinelinux.org/alpine/v3.21/main
+http://dl-cdn.alpinelinux.org/alpine/v3.21/community

update

apk update
apk cache download
apk cache -v sync

setup partition

Note

set it with your needs, I use entire disk in this docs

make sure first your ext4 kernel module is installed & loaded by typing

lsmod | grep ext4: check
modprobe ext4
lsmod | grep ext4: recheck again

the output is should be something like this

then we can move forward to ext4 partition generation

partitioning

install parted first using apk add parted, check your disk first using fdisk -l, for example there is /dev/sda

parted section

run parted /dev/sda, inside of parted is

unit s
mklabel [gpt/msdos]: I use msdos anyway
print: identify disk size by see on section something like Disk /dev/sda: 2097152s, there has 2097152s sector, so I use 2097152 - (2948 * 2), which is 2093056s
mkpart
- primary
- ext4
- 2048s
- 2093056s
  - q

resulted partition:

setup diskless install

warn: you need to enable ext4 kernel module or you'll get mount: mounting /dev/sda1 on /media/sda1 failed: Invalid argument when mounting

modprobe ext4
apk add e2fsprogs for mkfs.ext4 command
mkfs.ext4 /dev/sda1
mount /dev/sda1 /media/sda1: mount location should be located on /media, not /mnt

then, you can do normal setup-alpine, except for this thing

Which disk(s) would you like to use? (or '?' for help or 'none') [none] none
Enter where to store configs ('floppy', 'sda1', 'usb' or 'none') [sda1] sda1
Enter apk cache directory (or '?' or 'none') [/media/sda1/cache] /media/sda1/cache

LBU

run lbu commit

Run router-OS on qemu

on normal interface

qemu-system-x86_64 \
          -enable-kvm \
          -boot order=d \
          -cdrom chr-6.49.18.img \
          -drive file=chr.qcow2,format=qcow2 \
          -m 4G \
          -smp 4 \
          -net user,hostfwd=tcp::10022-:22,hostfwd=tcp::8291-:8291 \
          -net nic

setup linux bridge between RouterOS & Alpine linux hosts

this docs is second part of this

first thing that we need to do is create a copy of qcow2 alpine, so we do not need to re-install again. do not forget to change the hostname & password for easy identification

setup bridge

sudo ip link add br0-lan type bridge
sudo ip link add br1-lan type bridge

setup TAP

tap0 & tap1 connected to br0-lan tap2 & tap3 connected to br1-lan

sudo ip tuntap add dev tap0 mode tap
sudo ip tuntap add dev tap1 mode tap
sudo ip tuntap add dev tap2 mode tap
sudo ip tuntap add dev tap3 mode tap
sudo ip link set tap0 master br0-lan
sudo ip link set tap1 master br0-lan
sudo ip link set tap2 master br1-lan
sudo ip link set tap3 master br1-lan
sudo ip link set tap0 up
sudo ip link set tap1 up
sudo ip link set tap2 up
sudo ip link set tap3 up

launch mikrotik ISO

qemu-system-x86_64 \
      -enable-kvm \
      -boot order=d \
      -drive file=chr.qcow2,format=qcow2 \
      -m 256M \
      -smp 4 \
      -netdev tap,id=net0,ifname=tap0,script=no,downscript=no \
      -device virtio-net-pci,netdev=net0,mac=02:11:2a:3b:ff:c3  \
      -netdev tap,id=net1,ifname=tap2,script=no,downscript=no \
      -device virtio-net-pci,netdev=net1,mac=02:1a:f2:f1:21:b5  \
      -nographic \
      -net user,hostfwd=tcp::10022-:22,hostfwd=tcp::8291-:8291 \
      -net nic

based on topology, we know that

02:11:2a:3b:ff:c3 used by tap0 (in routerOS, bridge 0)
02:1a:f2:f1:21:b5 used by tap2 (in routerOS, bridge 1)
tap1 & tap3 used by alpine virtual machine

now run alpine iso

alpine (as pc-1)

qemu-system-x86_64 \
            -enable-kvm \
            -m 256 \
            -smp 4 \
            -cpu host \
            -drive file=pc-1.qcow2,format=qcow2 \
            -cdrom alpine-virt-3.21.3-x86_64.iso \
            -boot d \
            -nographic \
            -netdev tap,id=net0,ifname=tap1,script=no,downscript=no \
            -device virtio-net-pci,netdev=net0

alpine (as pc-3)

qemu-system-x86_64 \
            -enable-kvm \
            -m 256 \
            -smp 4 \
            -cpu host \
            -drive file=pc-2.qcow2,format=qcow2 \
            -cdrom alpine-virt-3.21.3-x86_64.iso \
            -boot d \
            -nographic \
            -netdev tap,id=net0,ifname=tap3,script=no,downscript=no \
            -device virtio-net-pci,netdev=net0

WinBox config

first, we need to see the configuration & which etherX by looking to spesific mac-addr, we find out that

[admin@MikroTik] > /interface print
Flags: D - dynamic, X - disabled, R - running, S - slave 
 #     NAME                                TYPE       ACTUAL-MTU L2MTU  MAX-L2MTU MAC-ADDRESS      
 0  R  ;;; wan
       ether1                              ether            1500                  52:54:00:12:34:56
 1  R  ether4                              ether            1500                  02:11:2A:3B:FF:C3
 2  R  ether5                              ether            1500                  02:1A:F2:F1:21:B5
[admin@MikroTik] >

next thing, lets configure our ip using winbox, this is ip configuration

DHCP server section

now, try run udhcpc -i eth0 on each vm

result on pc-1

result on pc-2

Linux iptables preview (try making a nat)

for example, iptables -t nat the t option from manpage is

          filter:
              This  is  the  default  table  (if  no -t option is passed). It contains the built-in
              chains INPUT (for packets destined to local  sockets),  FORWARD  (for  packets  being
              routed through the box), and OUTPUT (for locally-generated packets).

          nat:
              This  table  is consulted when a packet that creates a new connection is encountered.
              It consists of four built-ins: PREROUTING (for altering packets as soon as they  come
              in),  INPUT  (for  altering packets destined for local sockets), OUTPUT (for altering
              locally-generated packets before routing), and POSTROUTING (for altering  packets  as
              they are about to go out).  IPv6 NAT support is available since kernel 3.7.

          mangle:
              This table is used for specialized packet alteration.  Until kernel 2.4.17 it had two
              built-in chains: PREROUTING (for altering incoming packets before routing) and OUTPUT
              (for  altering locally-generated packets before routing).  Since kernel 2.4.18, three
              other built-in chains are also supported: INPUT (for packets coming into the box  it‐
              self),  FORWARD  (for altering packets being routed through the box), and POSTROUTING
              (for altering packets as they are about to go out).

          raw:
              This table is used mainly for configuring exemptions from connection tracking in com‐
              bination with the NOTRACK target.  It registers at the netfilter  hooks  with  higher
              priority and is thus called before ip_conntrack, or any other IP tables.  It provides
              the  following  built-in chains: PREROUTING (for packets arriving via any network in‐
              terface) and OUTPUT (for packets generated by local processes).

          security:
              This table is used for Mandatory Access Control (MAC) networking rules, such as those
              enabled by the SECMARK and CONNSECMARK targets.  Mandatory Access Control  is  imple‐
              mented by Linux Security Modules such as SELinux.  The security table is called after
              the filter table, allowing any Discretionary Access Control (DAC) rules in the filter
              table  to  take  effect before MAC rules.  This table provides the following built-in
              chains: INPUT (for packets coming into the box itself), OUTPUT (for altering locally-
              generated packets before routing), and FORWARD (for  altering  packets  being  routed
              through the box).

What kind of tables in iptables

according from archlinux wiki

iptables contains five tables:

raw is used only for configuring packets so that they are exempt from connection tracking.
filter is the default table, and is where all the actions typically associated with a firewall take place.
nat is used for network address translation (e.g. port forwarding).
mangle is used for specialized packet alterations.
security is used for Mandatory Access Control networking rules (e.g. SELinux -- see this article for more details).

In most common use cases, you will only use two of these: filter and nat. The other tables are aimed at complex configurations involving multiple routers and routing decisions and are in any case beyond the scope of these introductory remarks.

Tables & chain details

all iptables chain information can be gathered by this command

sudo iptables -t filter -L
sudo iptables -t nat -L
sudo iptables -t mangle -L
sudo iptables -t raw -L
sudo iptables -t security -L

this is list of all chain by corresponding table

table `raw`

PREROUTING
OUTPUT

table `nat`

PREROUTING
INPUT
OUTPUT
POSTROUTING

table `mangle`

PREROUTING
INPUT
FORWARD
OUTPUT
POSTROUTING

table `filter`

INPUT
FORWARD
OUTPUT

table `security`

INPUT
FORWARD
OUTPUT

we will focus on NAT section.

machine session

In iptables, packet often categorized as 4 different state, such

NEW
ESTABLISHED
RELATED
INVALID

this connection tracking is done by a special framework within the kernel called conntrack

command lists

this is some special iptables command collection

iptables --list-rules: show all ip rules
iptables --table nat --list --line-numbers: will useful if you want to delete spesific rule, i.e, duplicated rule
sudo iptables --table nat -D POSTROUTING 2 example: delete rule 2

create your own chain

iptables -N chain_name: eq: --new-chain
iptables -A chain_name -p icmp -j accept (example)

make changes permanent

sudo iptables-save > /etc/iptables/rules.v4

tables explanation

> sudo iptables --table nat --list
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  anywhere             anywhere            
MASQUERADE  all  --  anywhere             anywhere            
MASQUERADE  all  --  anywhere             anywhere            
MASQUERADE  all  --  anywhere             anywhere

target: what to do with packet. prot: abbr of protocol opt: abbr of options (AI says rarely used) source: its anywhere or explicit ip such 192.168.1.0/24 destination: same as source, can be network

show rules from X table

for example, there has

nat table, which contains PREROUTING, INPUT, OUTPUT, POSTROUTING
filter table contains main chain that used by iptables for packet filtering, which contains INPUT, FORWARD, OUTPUT

MORE deeper on X table

Table	Purpose	Chains it uses
`filter`	Default, handles packet filtering	`INPUT`, `OUTPUT`, `FORWARD`
`nat`	Network Address Translation	`PREROUTING`, `POSTROUTING`, `OUTPUT`
`mangle`	Packet modification (TTL, TOS, etc.)	All chains
`raw`	Pre-connection tracking processing	`PREROUTING`, `OUTPUT`
`security`	SELinux/LSM-based packet filtering	`INPUT`, `OUTPUT`, `FORWARD`

Linux QEMU network address translation

first all, set setup our interface

sudo ip tuntap add tap0 mode tap
sudo ip tuntap add tap1 mode tap (optional, if there has any more vm)
sudo ip link add br0-lan type bridge (our switch)
sudo ip addr add 192.168.12.1/24 dev br0-lan
sudo ip link set br0-lan up
sudo ip link set tap0 up, also with tap1 if needed

Run alpine guest hosts

qemu-system-x86_64 \
            -enable-kvm \
            -m 256 \
            -smp 4 \
            -cpu host \
            -drive file=pc-2.qcow2,format=qcow2 \
            -cdrom alpine-virt-3.21.3-x86_64.iso \
            -boot d \
            -nographic \
            -netdev tap,id=net0,ifname=tap0,script=no,downscript=no \
            -device virtio-net-pci,netdev=net0

Inside of guest VM

Your ip addr output might something like this

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
    link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5054:ff:fe12:3456/64 scope link 
       valid_lft forever preferred_lft forever

now set eth0 up by typing

ip link set eth0 up
ip addr add 192.168.12.2/24 dev eth0 (I manually setting up IP)

check ip route

pc-2:~# ip r
192.168.12.0/24 dev eth0 scope link  src 192.168.12.2

there is no route, run ip route add default via 192.168.12.1, check again with ip route

pc-2:~# ip r
default via 192.168.12.1 dev eth0 
192.168.12.0/24 dev eth0 scope link  src 192.168.12.2

setting up firewall

we want everyting from br0-lan is forwarded into wlan0, vice versa. in order to do that, we need NAT (network address translation)

here

check your nat table first

sudo iptables --table nat --list -v, make sure there is no

 2151  499K MASQUERADE  all  --  any    wlan0   anywhere             anywhere

now run

sudo iptables --table nat --append POSTROUTING --out-interface wlan0 -j MASQUERADE
sudo iptables -t filter -A FORWARD -i wlan0 -o br0-lan -m state --state RELATED,ESTABLISHED -j ACCEPT
sudo iptables -t filter -A FORWARD -i br0-lan -o wlan0 -j ACCEPT

QEMU block 8.8.8.8 from host

after this

I want this rule

allow other ICMP (i.e 1.1.1.1)
deny ICMP 8.8.8.8

iptables rule

suppose, vm ip is 192.168.12.2, so the iptables command is

sudo iptables -t filter -A FORWARD -s 192.168.12.2 -d 8.8.8.8 -p icmp -j DROP

note:

t: explicit table
a: table FORWARD
s: source from 192.168.12.2 (OUR VM)
d: destination: 8.8.8.8
p: protocol (which is ICMP)
j: jump

the problem

you may see ping 8.8.8.8 still success, let check iptables rule

sudo iptables -t filter -L FORWARD -n --line-numbers -v

the output

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination         
1    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
2    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
3    DROP       icmp --  192.168.12.2         8.8.8.8

see? when ping to 8.8.8.8, its stop at rule 2, which abort 3st rule, in order to fix it, we need re-order this

remote all rule

sudo iptables -t filter -F FORWARD

re-add

sudo iptables -t filter -A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
sudo iptables -t filter -A FORWARD -s 192.168.12.2 -d 8.8.8.8 -p icmp -j DROP
sudo iptables -t filter -A FORWARD -s 192.168.12.2/24 -j ACCEPT

Try re-run sudo iptables -t filter -L FORWARD -n --line-numbers

the order should be fixed

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination         
1               all  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
2    DROP       icmp --  192.168.12.2         8.8.8.8             
3    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0

two alpine guest VM, block custom ports

logic netfilter linux kernel

Logic NAT

misalkan kita ada virtual interface br0, dengan anggotanya tap0, tap1 lalu ingin forward semua traffic ke wlan0 (karna memang wlan0 tidak bisa di jadikan master dari x y z)

maka solusinya iptables

sudo iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE

logic nya

fill dulu tabel nat
append ke postrouting, jadi setelah packet di routing oleh linux kernel, ada 2 field, source addr (katakanlah ip tap0) lalu destination addr (misal 8.8.8.8)
nah, semisal tidak di apa apakan, packet sent ke 8.8.8.8, tapi 8.8.8.8 tidak bisa mengirim back packetnya karna source addr nya invalid (di stage ini, source ip nya adalah IP LAN)
maka, setelah linux kernel meng-routingkan packet nya, seketika setelah itu (postrouting), kita ganti source addrnya dengan source ip PUBLIK milik firewall, jadi 8.8.8.8 bisa tahu kemana packet harus dikirim kembali
didalam firewall, packet yang tadi diterima, direverse balik oleh nat, dan dikirim balik ke perangkat aslinya.
pakai MASQUERADE

logic forward data dari br0 ke wlan

sudo iptables -A FORWARD -i br0 -o wlan0 -j ACCEPT

important notes:

chain FORWARD: This chain is only used for packets that are not destined for the local machine, but are routed through it (i.e., from one interface to another).
In this case: a device connected to br0 (e.g., a VM or container) wants to access the internet via wlan0.

analoginya

[Device 192.168.100.2] ──> [br0 (Linux bridge)] ──> Linux router ──> [wlan0] ──> Internet

bisa juga -j diganti drop, maka aliran data akan terputus

logic forward dari wlan0 ke br0 (sebaliknya)

sudo iptables -A FORWARD -i wlan0 -o br0 -m state --state RELATED,ESTABLISHED -j ACCEPT

logic:

misal vm tap0 buat koneksi keluar, misal ke 8.8.8.8
dari tap0 -> br0 -> wlan0 di allow oleh rule sudo iptables -A FORWARD -i br0 -o wlan0 -j ACCEPT
ketika server mengirim respon balik, maka alurnya dari wlan0 ke br0, nah ini fungsi dari sudo iptables -A FORWARD -i wlan0 -o br0 -j ACCEPT

KENAPA harus pakai --state RELATED,ESTABLISHED karna kita butuh packet yg sudah estab duluan yg boleh lewat. ini akan menghindari jaringan luar (wlan0) mengakses br0 -> tap0 & tap1

resep nginx oleh pak hanif

cuplikan config nginx di server, cc @hansputera


server {
        listen 80; # tanpa server_default
        listen [::]:80;

        root /var/www/html/blablabla.fadev.org;
        
        index index.php index.html index.htm index.nginx-debian.html;

        server_name blablabla.fadev.org;

        location ~ \.php$ {
                include snippets/fastcgi-php.conf;

                fastcgi_pass unix:/run/php/php-fpm.sock;
        }
}

ip command

`ip addr`

ip addr is used to display and manage IP addresses assigned to network interfaces.

example:

ip addr
ip addr show
sudo ip addr add 192.168.1.100/24 dev eth0
sudo ip addr del 192.168.1.100/24 dev eth0

anything about ip assigment

`ip link`

ip link is used to display and manage network interfaces (links), focusing on layer 2 (Ethernet, loopback, etc.), not IP addresses.

example:

ip link
sudo ip link set dev eth0 up
sudo ip link set dev eth0 down
sudo ip link set dev eth0 mtu 1400

anyting about interface

`ip tuntap`

ip tuntap manages TUN/TAP virtual network interfaces, which provide a way for user-space programs to interact with network packets as if they were network devices.

TUN: Simulates a point-to-point device (layer 3), used for routing IP packets (e.g., VPN tunnels).
TAP: Simulates an Ethernet device (layer 2), used for bridging Ethernet frames (e.g., virtual switches, VMs).

example:

sudo ip tuntap add mode tap dev tap0
sudo ip tuntap add mode tun dev tun0
sudo ip tuntap del mode tap dev tap0

`ip route`

IP routing is the process by which data packets are directed from their source to their destination across interconnected networks using Internet Protocol (IP). This process is fundamental to the operation of the Internet and all IP-based networks.

View routing table:

ip route

add route

ip route add <destination> via <gateway> dev <interface>

example:

pc-1:~# ip route add default via 192.168.15.1 dev eth0
pc-1:~# ip route
default via 192.168.15.1 dev eth0 
192.168.15.0/24 dev eth0 scope link  src 192.168.15.2

delete route

sudo ip route del <destination> [via <gateway>] [dev <interface>]

example:

pc-1:~# ip route delete default via 192.168.15.1 dev eth0
pc-1:~# ip r
192.168.15.0/24 dev eth0 scope link  src 192.168.15.2 
pc-1:~#

qemu default network

The default gateway in QEMU when using the default user-mode networking (SLIRP) is typically 10.0.2.2 on the network 10.0.2.0/24. QEMU creates a virtual NAT network where the guest is assigned an IP like 10.0.2.15 by DHCP, and the gateway for the guest to reach outside networks is 10.0.2.2. This gateway is implemented inside QEMU and handles routing and NAT for the guest's traffic.

Default network: 10.0.2.0/24
Default gateway: 10.0.2.2
Default guest IP (DHCP assigned): 10.0.2.15
Default DNS server in guest: 10.0.2.3

This setup requires no special host configuration or root privileges and is the default networking mode for QEMU user mode networking.

additional

mikrotik firewall

each firewall module (called as table in iptables) has its own predefined chain

raw: dipakai sebelum data dilihat oleh conntrack, NAT, masuk routing, dll. intinya di sinilah proses early sebelum benar2 di proses. karna belum sampai ke conntrack, maka akan sangat fast, dipakai untuk mitigasi ddos

karna statenya ada sebelum conntrack, maka tidak ada status kayak estab, dll, tapi sangat fast + low cpu, only raw matching

Karna raw table melihat packet sebelum NAT, maka
- Destination IPs are still the original IPs from the client
- Source IPs haven't been changed by masquerade or src-nat yet
chain chain nya
- prerouting: packet yang datang dari luar just sebelum masuk conntrack
- output: packet yang digenerate oleh aplikasi, juga statenya sebelum di track oleh conntrack
contoh
- /ip firewall raw add chain=prerouting src-address=8.8.8.8 action=drop
tambahan:

letak raw di packet flow
```
IN → RAW → MANGLE (pre) → CONNECTION TRACKING → NAT (dstnat) → FILTER (input/forward) → MANGLE (post) → NAT (srcnat) → OUT
```
filter
- input
- forward
- output
mangle
- prerouting
- input
- forward
- output
- postrouting
nat
- srcnat
- dstnat

chains

RouterOS consist of a few default chains. These chains allow you to filter packets at various points:

The PREROUTING chain: Rules in this chain apply to packets as they just arrive on the network interface. This chain is present in the nat, mangle and raw tables.
The INPUT chain: Rules in this chain apply to packets just before they’re given to a local process. This chain is present in the mangle and filter tables.
The OUTPUT chain: The rules here apply to packets just after they’ve been produced by a process. This chain is present in the raw, mangle, nat, and filter tables.
The FORWARD chain: The rules here apply to any packets that are routed through the current host. This chain is only present in the mangle and filter tables.
The POSTROUTING chain: The rules in this chain apply to packets as they just leave the network interface. This chain is present in the nat and mangle tables.

bring wlan0 connectivity to the network namespace linux & test routerOS

setup network namespace

sudo ip netns add firefoxns
sudo ip netns exec firefoxns ip link add br0-lan type bridge
sudo ip netns exec firefoxns ip tuntap add tap0 mode tap
sudo ip netns exec firefoxns ip tuntap add tap1 mode tap
sudo ip netns exec firefoxns ip tuntap add tap2 mode tap
sudo ip netns exec firefoxns ip link set dev tap0 master br0-lan
sudo ip netns exec firefoxns ip link set dev tap1 master br0-lan
sudo ip netns exec firefoxns ip link set dev tap2 master br0-lan

enable the network interface (inside netns)

sudo ip netns exec firefoxns ip link set br0-lan up
sudo ip netns exec firefoxns ip link set tap0 up
sudo ip netns exec firefoxns ip link set tap1 up
sudo ip netns exec firefoxns ip link set tap2 up

create virtual eth pair between host & netns

sudo ip link add veth0 type veth peer name veth0-peer
sudo ip link set veth0-peer netns firefoxns

assign ip for virtual eth & virtual eth peer

sudo ip addr add 10.200.1.1/24 dev veth0
sudo ip netns exec firefoxns ip addr add 10.200.1.2/24 dev veth0-peer
sudo ip link set veth0 up
sudo ip netns exec firefoxns ip link set veth0 up

setup routing table

sudo ip netns exec firefoxns ip route add default via 10.200.1.1

setup ip forwarding & network address translation

sudo sysctl -w net.ipv4.ip_forward=1
sudo iptables -t nat -A POSTROUTING -s 10.200.1.0/24 -o wlan0 -j MASQUERADE

test

sudo ip netns exec ping 1.1.1.1

now, run routeros & alpine inside of netns

routeros

sudo ip netns exec firefoxns qemu-system-x86_64 \
            -enable-kvm \
            -m 256 \
            -smp 4 \
            -cpu host \
            -drive file=chr.qcow2,format=qcow2 \
            -boot d \
            -nographic \
            -netdev tap,id=net0,ifname=tap0,script=no,downscript=no \
            -device virtio-net-pci,netdev=net0,mac=02:aa:bb:cc:dd:ee \
            -net user,hostfwd=tcp::8291-:8291 \
            -net nic

alpine

sudo ip netns exec firefoxns qemu-system-x86_64 \
            -enable-kvm \
            -m 256 \
            -smp 4 \
            -cpu host \
            -drive file=pc-2.qcow2,format=qcow2 \
            -cdrom alpine-virt-3.21.3-x86_64.iso \
            -boot d \
            -nographic \
            -netdev tap,id=net0,ifname=tap2,script=no,downscript=no \
            -device virtio-net-pci,netdev=net0,mac=$(randommac)

network namespace + NAT notes

ip a
sudo ip link add veth0 type veth peer name veth0-peer
sudo ip addr add 10.200.1.1/24 dev veth0
sudo ip netns add firefoxns
sudo ip link set veth0-peer netns firefoxns
sudo sysctl -w net.ipv4.ip_forward=1
sudo iptables -t nat -A POSTROUTING -s 10.200.1.0/24 -o wlan0 -j MASQUERADE
sudo ip netns exec firefoxns ip addr add 10.200.1.2/24 dev veth0-peer
sudo ip netns exec firefoxns ip addr add 127.0.0.1/8 dev lo
sudo ip netns exec firefoxns ip link set lo up
sudo ip netns exec firefoxns ip link set veth0-peer up
sudo ip netns exec firefoxns ip route add default via 10.200.1.1
sudo ip netns exec firefoxns ip link add br0-lan type bridge
sudo ip netns exec firefoxns ip tuntap add tap0 mode tap
sudo ip netns exec firefoxns ip tuntap add tap1 mode tap
sudo ip netns exec firefoxns ip tuntap add tap2 mode tap
sudo ip netns exec firefoxns ip link set dev tap0 master br0-lan
sudo ip netns exec firefoxns ip link set dev tap1 master br0-lan
sudo ip netns exec firefoxns ip link set dev tap2 master br0-lan
sudo ip netns exec firefoxns ip link set dev tap0 up
sudo ip netns exec firefoxns ip link set dev tap1 up
sudo ip netns exec firefoxns ip link set dev tap2 up
sudo ip netns exec firefoxns ip link set dev br0-lan up
sudo ip netns exec firefoxns qemu-system-x86_64 -enable-kvm -m 256 -smp 4 -cpu host -drive file=chr.qcow2,format=qcow2 -boot d -nographic -netdev tap,id=net0,ifname=tap0,script=no,downscript=no -device virtio-net-pci,netdev=net0,mac=02:aa:bb:cc:dd:ee -net user,hostfwd=tcp::8291-:8291 -net nic
sudo ip netns exec firefoxns qemu-system-x86_64 -enable-kvm -m 256 -smp 4 -cpu host -drive file=pc-1.qcow2,format=qcow2 -cdrom alpine-virt-3.21.3-x86_64.iso -boot d -nographic -netdev tap,id=net0,ifname=tap1,script=no,downscript=no -device virtio-net-pci,netdev=net0,mac=$(randommac)

randommac source code

use rand::Rng;

fn main() {
    let mut rng = rand::thread_rng();
    let mac = format!(
        "52:54:00:{:02x}:{:02x}:{:02x}",
        rng.gen::<u8>(),
        rng.gen::<u8>(),
        rng.gen::<u8>()
    );
    println!("{}", mac);
}

deploy Docmost on server

please read this doc first, https://docmost.com/docs/installation

docker compose config

version: "3"

services:
  docmost:
    image: docmost/docmost:latest
    depends_on:
      - db
      - redis
    environment:
      APP_URL: "http://localhost:3000"
      APP_SECRET: "REPLACE_WITH_LONG_SECRET"
      DATABASE_URL: "postgresql://docmost:STRONG_DB_PASSWORD@db:5432/docmost?schema=public"
      REDIS_URL: "redis://redis:6379"
    ports:
      - "3000:3000"
    restart: unless-stopped
    volumes:
      - docmost:/app/data/storage

  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: docmost
      POSTGRES_USER: docmost
      POSTGRES_PASSWORD: STRONG_DB_PASSWORD
    restart: unless-stopped
    volumes:
      - db_data:/var/lib/postgresql/data

  redis:
    image: redis:7.2-alpine
    restart: unless-stopped
    volumes:
      - redis_data:/data

volumes:
  docmost:
  db_data:
  redis_data:

replace your STRONG_DB_PASSWORD with openssl rand -hex 10, and REPLACE_WITH_LONG_SECRET with openssl rand -hex 32, and APP_URL, for example, in my case is https://docmost.fadev.org

run docker compose run

setup reverse proxy (nginx)

go to /etc/nginx/sites-enabled, and create a clone with cp default docmost.fadev.org

fill it like this

server_name docmost.fadev.org;

location / {
	# First attempt to serve request as file, then
	# as directory, then fall back to displaying a 404.
	#try_files $uri $uri/ =404;

	proxy_pass http://127.0.0.1:3000;
}

location /socket.io/ {
	proxy_pass http://127.0.0.1:3000;
	proxy_http_version 1.1;
	proxy_set_header Upgrade $http_upgrade;
	proxy_set_header Connection "upgrade";
	proxy_set_header Host $host;
}

location /collab {
	proxy_pass http://127.0.0.1:3000;
	proxy_http_version 1.1;
	proxy_set_header Upgrade $http_upgrade;
	proxy_set_header Connection "upgrade";
	proxy_set_header Host $host;
}

HTTPS

run

sudo apt install certbot python3-certbot-nginx
sudo certbot --nginx -d docmost.fadev.org

Firewall

allow port 443

fix vthxxxxxxxx is not connected to docker0

consider

> ip  a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp2s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 2c:4d:54:c6:f2:0c brd ff:ff:ff:ff:ff:ff
    altname enx2c4d54c6f20c
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f0:03:8c:66:9c:21 brd ff:ff:ff:ff:ff:ff
    inet 192.168.105.162/24 metric 1024 brd 192.168.105.255 scope global dynamic wlan0
       valid_lft 2920sec preferred_lft 2920sec
    inet 192.168.105.163/24 brd 192.168.105.255 scope global secondary dynamic noprefixroute wlan0
       valid_lft 2921sec preferred_lft 2921sec
    inet6 fe80::ee67:75f9:fa9c:2561/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 86:1e:e0:72:f4:8d brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::841e:e0ff:fe72:f48d/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
17: veth914db4f@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether d2:02:61:f4:12:d0 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::d002:61ff:fef4:12d0/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

you can see, veth914db4f is not connected to br0. let's try hard

run image (explicitly) as bridge network: docker run --network bridge -it --rm -p 127.0.0.1:8000:4000 php-test-server (this is nothing happen)
force connect: docker network connect bridge 2cf520235e6a (this is show Error response from daemon: endpoint with name mystifying_yonath already exists in network bridge, but ip a says vethxxxx is not master to anyone). you can try disconnect & connect again

also

docker inspect -f '{{json .NetworkSettings.Networks}}' 2cf520235e6a | jq
{
  "bridge": {
    "IPAMConfig": null,
    "Links": null,
    "Aliases": null,
    "MacAddress": "5a:2d:5d:7c:c1:86",
    "DriverOpts": null,
    "GwPriority": 0,
    "NetworkID": "08f151565c95ba052f682c7560e55199e2d75f3d2348af8f98a9711e9294b3fd",
    "EndpointID": "0522c15ad18fbae05071a7d5e09944b7777deaafd3490082cac237fdfcd14f0c",
    "Gateway": "172.17.0.1",
    "IPAddress": "172.17.0.2",
    "IPPrefixLen": 16,
    "IPv6Gateway": "",
    "GlobalIPv6Address": "",
    "GlobalIPv6PrefixLen": 0,
    "DNSNames": null
  }
}

the medicine: https://forums.docker.com/t/no-connection-to-the-bridge-by-default/134619/8

sudo systemctl stop systemd-networkd.service                                                    
sudo systemctl disable systemd-networkd.service                                                        
sudo systemctl stop systemd-networkd.socket                                                            
sudo systemctl disable systemd-networkd.socket
sudo systemctl start NetworkManager
sudo systemctl enable NetworkManager

php-fpm config

file ini biasanya ada di

/usr/local/etc/php-fpm.d/www.conf (di docker php-fpm)
/etc/php/8.3/fpm/pool.d/www.conf (di debian bullseye)

intinya, tidak pasti haha.

config structure

general structure nya seperti ini kalau ';' dihilangkan

[www]
user = www-data
group = www-data
listen = /run/php/php8.3-fpm.sock
listen.owner = www-data
listen.group = www-data
pm = dynamic
pm.max_children = 5
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3

config global (WIP)

config untuk pool (each)

best read https://www.php.net/manual/en/install.fpm.configuration.php

[pool_name] intinya ini unique, tidak ada yang sama
user / group
- user: dimana child process akan di run (as user?)
- group: liat /etc/passwd
listen
- listen: dimana daemon akan listen yg nanti akan di reverse proxy oleh nginx, nilai defaultnya biasanya /run/php/php8.3-fpm.sock atau 127.0.0.1:9000, contoh:
  - ip.add.re.ss:port
  - [ip:6:addr:ess]:port
  - port
  - /path/to/unix/socket
- listen.backlog: jumlah queue pending connection yang bisa di hold
- listen.owner: configure ke mana kah /run/php/php8.3-fpm.sock itu ownernya
- listen.group: same as listen.owner
- listen.mode: nomor permission si /run/php/php8.3-fpm.sock listen
- listen.acl_users: WIP
- listen.acl_groups: WIP
- listen.allowed_clients: set dari mana FCGI boleh diakses, di kasus nginx, mostly 127.0.0.1, except docker, dia pakai br-xxxxxx yang ip nya pasti bukan 127.0.0.1
- listen.setfib: WIP

systemd config template

root location: /etc/systemd/system/<YOUR_SERVICE>.service

template

[Unit]
Description=Random service
After=network.target
StartLimitIntervalSec=0

[Service]
Type=simple
Restart=always
RestartSec=1
User=root
ExecStart=/usr/bin/randomprogram -c /etc/randomprogram/other.toml

[Install]
WantedBy=multi-user.target

Database

psql command

General

\bind [PARAM]... set query parameters
\copyright show PostgreSQL usage and distribution terms
\crosstabview [COLUMNS] execute query and display result in crosstab
\errverbose show most recent error message at maximum verbosity
\g [(OPTIONS)] [FILE] execute query (and send result to file or |pipe); \g with no arguments is equivalent to a semicolon
\gdesc describe result of query, without executing it
\gexec execute query, then execute each value in its result
\gset [PREFIX] execute query and store result in psql variables
\gx [(OPTIONS)] [FILE] as \g, but forces expanded output mode
\q quit psql
\watch [[i=]SEC] [c=N] [m=MIN] execute query every SEC seconds, up to N times, stop if less than MIN rows are returned

Help

\? [commands] show help on backslash commands
\? options show help on psql command-line options
\? variables show help on special variables
\h [NAME] help on syntax of SQL commands, * for all commands

Query Buffer

\e [FILE] [LINE] edit the query buffer (or file) with external editor
\ef [FUNCNAME [LINE]] edit function definition with external editor
\ev [VIEWNAME [LINE]] edit view definition with external editor
\p show the contents of the query buffer
\r reset (clear) the query buffer
\s [FILE] display history or save it to file
\w FILE write query buffer to file

Input/Output

\copy ... perform SQL COPY with data stream to the client host
\echo [-n] [STRING] write string to standard output (-n for no newline)
\i FILE execute commands from file
\ir FILE as \i, but relative to location of current script
\o [FILE] send all query results to file or |pipe
\qecho [-n] [STRING] write string to \o output stream (-n for no newline)
\warn [-n] [STRING] write string to standard error (-n for no newline)

Conditional

\if EXPR begin conditional block
\elif EXPR alternative within current conditional block
\else final alternative within current conditional block
\endif end conditional block

Informational

(options: S = show system objects, + = additional detail)

\d[S+] list tables, views, and sequences
\d[S+] NAME describe table, view, sequence, or index
\da[S] [PATTERN] list aggregates
\dA[+] [PATTERN] list access methods
\dAc[+] [AMPTRN [TYPEPTRN]] list operator classes
\dAf[+] [AMPTRN [TYPEPTRN]] list operator families
\dAo[+] [AMPTRN [OPFPTRN]] list operators of operator families
\dAp[+] [AMPTRN [OPFPTRN]] list support functions of operator families
\db[+] [PATTERN] list tablespaces
\dc[S+] [PATTERN] list conversions
\dconfig[+] [PATTERN] list configuration parameters
\dC[+] [PATTERN] list casts
\dd[S] [PATTERN] show object descriptions not displayed elsewhere
\dD[S+] [PATTERN] list domains
\ddp [PATTERN] list default privileges
\dE[S+] [PATTERN] list foreign tables
\des[+] [PATTERN] list foreign servers
\det[+] [PATTERN] list foreign tables
\deu[+] [PATTERN] list user mappings
\dew[+] [PATTERN] list foreign-data wrappers
\df[anptw][S+] [FUNCPTRN [TYPEPTRN ...]] list [only agg/normal/procedure/trigger/window] functions
\dF[+] [PATTERN] list text search configurations
\dFd[+] [PATTERN] list text search dictionaries
\dFp[+] [PATTERN] list text search parsers
\dFt[+] [PATTERN] list text search templates
\dg[S+] [PATTERN] list roles
\di[S+] [PATTERN] list indexes
\dl[+] list large objects, same as \lo_list
\dL[S+] [PATTERN] list procedural languages
\dm[S+] [PATTERN] list materialized views
\dn[S+] [PATTERN] list schemas
\do[S+] [OPPTRN [TYPEPTRN [TYPEPTRN]]] list operators
\dO[S+] [PATTERN] list collations
\dp[S] [PATTERN] list table, view, and sequence access privileges
\dP[itn+] [PATTERN] list [only index/table] partitioned relations [n=nested]
\drds [ROLEPTRN [DBPTRN]] list per-database role settings
\drg[S] [PATTERN] list role grants
\dRp[+] [PATTERN] list replication publications
\dRs[+] [PATTERN] list replication subscriptions
\ds[S+] [PATTERN] list sequences
\dt[S+] [PATTERN] list tables
\dT[S+] [PATTERN] list data types
\du[S+] [PATTERN] list roles
\dv[S+] [PATTERN] list views
\dx[+] [PATTERN] list extensions
\dX [PATTERN] list extended statistics
\dy[+] [PATTERN] list event triggers
\l[+] [PATTERN] list databases
\sf[+] FUNCNAME show a function's definition
\sv[+] VIEWNAME show a view's definition
\z[S] [PATTERN] same as \dp

Large Objects

\lo_export LOBOID FILE write large object to file
\lo_import FILE [COMMENT] read large object from file
\lo_list[+] list large objects
\lo_unlink LOBOID delete a large object

Formatting

\a toggle between unaligned and aligned output mode
\C [STRING] set table title, or unset if none
\f [STRING] show or set field separator for unaligned query output
\H toggle HTML output mode (currently off)
\pset [NAME [VALUE]] set table output option (border|columns|csv_fieldsep|expanded|fieldsep| fieldsep_zero|footer|format|linestyle|null| numericlocale|pager|pager_min_lines|recordsep| recordsep_zero|tableattr|title|tuples_only| unicode_border_linestyle|unicode_column_linestyle| unicode_header_linestyle|xheader_width)
\t [on|off] show only rows (currently off)

\T [STRING] set HTML tag attributes, or unset if none

\x [on|off|auto] toggle expanded output (currently off)

Connection

\c[onnect] {[DBNAME|- USER|- HOST|- PORT|-] | conninfo} connect to new database (currently "fadev_infra")
\conninfo display information about current connection
\encoding [ENCODING] show or set client encoding
\password [USERNAME] securely change the password for a user

Operating System

\cd [DIR] change the current working directory
\getenv PSQLVAR ENVVAR fetch environment variable
\setenv NAME [VALUE] set or unset environment variable
\timing [on|off] toggle timing of commands (currently off)
! [COMMAND] execute command in shell or start interactive shell

Variables

\prompt [TEXT] NAME prompt user to set internal variable
\set [NAME [VALUE]] set internal variable, or list all if no parameters
\unset NAME unset (delete) internal variable

psql snippet

psql

\df *.* Lists all relations
\l Lists all database

command

DROP TABLE table_name CASCADE; used when you want to remove all object depend on them.

PostgreSQL database creation flow

in order to create database (for safety, I use postgres account). I use this step.

first, we need to login as postgres user, use this sudo -u postgres psql, then do

CREATE DATABASE EXAMPLE_DB;

this is optional

CREATE USER EXAMPLE_USER WITH ENCRYPTED PASSWORD 'Sup3rS3cret';

grant

GRANT ALL PRIVILEGES ON DATABASE EXAMPLE_DB TO EXAMPLE_USER;

now, change the state using \c EXAMPLE_DB postgres, execute this command

GRANT ALL ON SCHEMA public TO EXAMPLE_USER;

then, change ownership

ALTER DATABASE my_database OWNER TO my_database_user;

notes: https://stackoverflow.com/questions/67276391/why-am-i-getting-a-permission-denied-error-for-schema-public-on-pgadmin-4

how to use SQLX cli tools for db migration such laravel migrations

let's meet with this crate, https://crates.io/crates/sqlx-cli, this tools allow you to use db migration such as laravel migration tools;

first, let's create env variable

export DATABASE_URL=postgres://fadhil_riyanto:<your_pass>@localhost/<target_db>

then, the things that we do next is only

sqlx migrate add -r <name>

running

sqlx migrate run

Postgresql internal (server)

this thing explain the internal of pgsql system (not the source code I mean)

difference between catalog vs schema

key point:

schema is folder like, grouping table together, can be created using

CREATE SCHEMA my_catalog;
CREATE TABLE my_catalog.metadata_table (...);

catalog is system schema, can't be created manually, part of internal pgsql system

view is a resulted table from XYZ queries, example

CREATE VIEW active_users AS 
	SELECT id, name, 
	FROM
		users 
	WHERE
		active = true;

then query it

SELECT * FROM active_users;

pg_catalog inside

this is some stuff inside of pg catalog, you can do that using \dt pg_catalog.*

pg_aggregate: Stores information about aggregate functions (like SUM, AVG, etc.).
pg_am: Lists access methods for indexes (e.g., btree, hash).
pg_amop: Defines operators used in access methods.
pg_amproc: Defines support functions used in access methods.
pg_attrdef: Stores default values for columns.
pg_attribute: Contains column definitions for all tables.
pg_auth_members: Shows role memberships (who is a member of what).
pg_authid: Stores user/role definitions (superuser access required).
pg_cast: Contains rules for casting between data types.
pg_class: Contains all table-like objects (tables, views, indexes, sequences, etc.).
pg_collation: Defines collations (rules for string comparison).
pg_constraint: Stores constraints like PRIMARY KEY, UNIQUE, CHECK, and FOREIGN KEY.
pg_conversion: Defines character set conversions.
pg_database: Stores information about each database in the cluster.
pg_db_role_setting: Stores per-user/per-database configuration settings (GUCs).
pg_default_acl: Defines default privileges for newly created objects.
pg_depend: Stores dependency relationships between database objects.
pg_description: Stores comments/descriptions on database objects.
pg_enum: Stores values for ENUM data types.
pg_event_trigger: Stores event trigger definitions (triggers on DDL commands).
pg_extension: Tracks installed extensions (like uuid-ossp, pgcrypto, etc.).
pg_foreign_data_wrapper: Stores definitions of foreign data wrappers (FDW).
pg_foreign_server: Stores foreign servers used by FDWs.
pg_foreign_table: Stores metadata for foreign tables.
pg_index: Contains metadata about indexes (e.g., indexed columns).
pg_inherits: Stores table inheritance relationships.
pg_init_privs: Records original privileges on built-in objects.
pg_language: Stores information about supported procedural languages.
pg_largeobject: Stores the actual data of large objects (blobs).
pg_largeobject_metadata: Stores metadata about large objects.
pg_namespace: Lists all schemas in the database (IMPORTANT)
pg_opclass: Stores index operator classes (how a datatype is indexed).
pg_operator: Stores SQL operators (like =, <, +, etc.).
pg_opfamily: Groups related operator classes.
pg_parameter_acl: Stores access control for configuration parameters (PostgreSQL 16+).
pg_partitioned_table: Stores metadata for partitioned tables.
pg_policy: Stores row-level security policies.
pg_proc: Contains all function and procedure definitions.
pg_publication: Stores logical replication publications.
pg_publication_namespace: Links publications to schemas.
pg_publication_rel: Links publications to individual tables.
pg_range: Stores definitions of range types (e.g., int4range).
pg_replication_origin: Tracks origins for logical replication.
pg_rewrite: Stores query rewrite rules (used in views, rules).
pg_seclabel: Stores security labels for database objects.
pg_sequence: Contains metadata for sequence generators.
pg_shdepend: Tracks dependencies involving shared objects (like roles, databases).
pg_shdescription: Stores comments on shared objects.
pg_shseclabel: Stores security labels on shared objects.
pg_statistic: Stores planner statistics for columns.
pg_statistic_ext: Stores extended planner statistics (multi-column, NDV, etc.).
pg_statistic_ext_data: Contains actual values for extended statistics.
pg_subscription: Defines logical replication subscriptions.
pg_subscription_rel: Lists tables included in subscriptions.
pg_tablespace: Lists all tablespaces (disk locations for data).
pg_transform: Stores type transformation functions for procedural languages.
pg_trigger: Stores triggers on tables.
pg_ts_config: Stores full-text search configurations.
pg_ts_config_map: Maps text search config tokens to dictionaries.
pg_ts_dict: Stores text search dictionaries.
pg_ts_parser: Defines tokenizers for full-text search.
pg_ts_template: Defines templates for building text search dictionaries.
pg_type: Stores all data types (built-in, custom, enum, composite). (IMPORTANT)
pg_user_mapping: Maps users to foreign servers.

pg_catalog.pg_database details

this docs can be found in https://www.postgresql.org/docs/16/catalog-pg-database.html

Column Name	Type	Description
`datname`	`name`	The name of the database.
`datdba`	`oid`	The OID of the role (user) that owns the database. Use `pg_get_userbyid(datdba)` to resolve it.
`encoding`	`int`	The character encoding of the database (e.g., UTF8 = 6). Use `pg_encoding_to_char(encoding)` to get the name.
`datlocprovider`	`char`	Locale provider used (`c` = libc, `i` = ICU).
`datistemplate`	`bool`	If `true`, the database can be used as a template for `CREATE DATABASE ... TEMPLATE`.
`datallowconn`	`bool`	If `false`, connections to this database are not allowed (except by superusers).
`datconnlimit`	`int`	The maximum number of concurrent connections allowed (-1 = no limit).
`datlastsysoid`	`oid`	The last system OID used in this database at creation (mainly historical).
`datfrozenxid`	`xid`	The transaction ID at which all tuples are known to be frozen (related to VACUUM).
`datminmxid`	`xid`	The minimum multixact ID that is still considered potentially unfrozen.
`dattablespace`	`oid`	OID of the default tablespace for the database. Use `pg_tablespace` to resolve.
`datcollate`	`name`	LC_COLLATE setting (how strings are sorted).
`datctype`	`name`	LC_CTYPE setting (how character classification works).
`daticulocale`	`text`	ICU locale (used if `datlocprovider = 'i'`).
`datcollversion`	`text`	Version of the collation used (important for collation versioning with ICU).
`datacl`	`aclitem[]`	Access privileges (GRANTs), stored as an array of ACL items.

Fadhil Books & Notes