# Asynchronous DNS

## Introduction

Lws now features optional asynchronous, ie, nonblocking recursive DNS
resolution done on the event loop, enable `-DLWS_WITH_SYS_ASYNC_DNS=1`
at cmake to build it in.

## Description

The default libc name resolution is via libc `getaddrinfo()`, which is
blocking, possibly for quite long periods (seconds).  If you are
taking care about latency, but want to create outgoing connections,
you can't tolerate this exception from the rule that everything in
lws is nonblocking.

Lws' asynchronous DNS resolver creates a caching name resolver
that directly queries the configured nameserver itself over UDP,
from the event loop.

It supports both ipv4 / A records and ipv6 / AAAA records (see later
for a description about how).  One server supported over UDP :53,
and the nameserver is autodicovered on linux, windows, and freertos.

Other features

 - lws-style paranoid response parsing
 - random unique tid generation to increase difficulty of poisoning
 - it's really integrated with the lws event loop, it does not spawn
   threads or use the libc resolver, and of course no blocking at all
 - platform-specific server address capturing (from /etc/resolv.conf
   on linux, windows apis on windows)
 - LRU caching
 - piggybacking (multiple requests before the first completes go on
    a list on the first request, not spawn multiple requests)
 - observes TTL in cache
 - TTL and timeout use `lws_sul` timers on the event loop
 - Uses CNAME resolution inside the same response if present, otherwise
   recurses to resolve the CNAME (up to 3 deep)
 - ipv6 pieces only built if cmake `LWS_IPV6` enabled

## Api

If enabled at cmake, the async DNS implementation is used automatically
for lws client connections.  It's also possible to call it directly, see
the api-test-async-dns example for how.

The Api follows that of `getaddrinfo()` but results are not created on
the heap.  Instead a single, const cached copy of the addrinfo struct
chain is reference-counted, with `lws_async_dns_freeaddrinfo()` provided
to deduct from the reference count.  Cached items with a nonzero
reference count can't be destroyed from the cache, so it's safe to keep
a pointer to the results and iterate through them.

## Dealing with IPv4 and IPv6

DNS is a very old standard that has some quirks... one of them is that
multiple queries are not supported in one packet, even though the protocol
suggests it is.  This creates problems on ipv6 enabled systems, where
it may prefer to have AAAA results, but the server may only have A records.

To square the circle, for ipv4 only systems (`LWS_IPV6=0`) the resolver
requests only A records.  For ipv6-capable systems, it always requests
first A and then immediately afterwards AAAA records.

To simplify the implementation, the tid b0 is used to differentiate
between A (b0 = 0) and AAAA (b0 = 1) requests and responses using the
same query body.

The first response to come back is parsed, and a cache entry made...
it leaves a note in the query about the address of the last `struct addrinfo`
record.  When the second response comes, a second allocation is made,
but not added to the logical cache... instead it's chained on to the
first cache entry and the `struct addrinfo` linked-list from the
first cache entry is extended into the second one.  At the time the
second result arrives, the query is destroyed and the cached results
provided on the result callback.

## Recursion

Where CNAMEs are returned, DNS servers may take two approaches... if the
CNAME is also resolved by the same server and so it knows what it should
resolve to, it may provide the CNAME resolution in the same response
packet.

In the case the CNAME is actually resolved by a different name server,
the server with the CNAME does not have the information to hand to also
resolve the CNAME in the same response.  So it just leaves it for the
client to sort out.

The lws implementation can deal with both of these, first it "recurses"
(it does not recurse on the process stack but uses its own manual stack)
to look for results in the same packet that told it about the CNAME.  If
there are no results, it resets the query to look instead for the CNAME,
and restarts it.  It allows this to happen for 3 CNAME deep.

At the end, either way, the cached result is set using the original
query name and the results from the last CNAME in the chain.

# Some notes for the windows jungle

This was how I compiled libwebsockets in windows March 2020.

## OpenSSL

### Installing prebuilt libs

I used the 1.1.1d (the latest) libs from here, as recommended on the OpenSSL site

[overbyte.eu](https:..wiki.overbyte.eu/wiki/index.php/ICS_Download#Download_OpenSSL_Binaries_.28required_for_SSL-enabled_components.29)

I had to use procmon64 (windows' strace) to establish that these libraries are
looking for a cert bundle at "C:\Program Files\Common Files\SSL\cert.pem"... it's not
included in the zip file from the above, so...

### Installing a cert bundle

You can get a trusted cert bundle from here

[drwetter/testssl cert bundle](https://raw.githubusercontent.com/drwetter/testssl.sh/3.1dev/etc/Microsoft.pem)

Save it into `C:\Program Files\Common Files\SSL\cert.pem` where openssl will be able to see it.

### Installing cmake

CMake have a windows installer thing downloadable from here

[cmake](https://cmake.org/download/)

after that you can use `cmake` from the terminal OK.

### Installing git

Visit the canonical git site to download their windows installer thing

[git](https://git-scm.com/download/win)

after that `git` from the terminal is working.

### Install the free "community" visual studio

You can do this through "windows store" by searching for "visual studio"

I installed as little as possible, we just want the C "C++" tools.

It still wouldn't link without the "mt" helper tool from the
huge windows SDK, so you have to install GB of that as well.

### Building

Somehow windows cmake seems slightly broken, some of the plugins and
examples are conditional on `if (NOT WIN32)`, but it configures them
anyway.  For this reason (it seems "only", it worked when I commented the
cmake entries for the related plugins) `-DLWS_WITH_MINIMAL_EXAMPLES=1`

Instead I followed how appveyor builds the stuff in CI... clone libwebsockets then

```
> git clone https://libwebsockets.org/repo/libwebsockets
> cd libwebsockets
> mkdir build
> cd build
> cmake ..
> cmake --build . --config DEBUG
```

Installing requires admin privs, I opened a second cmd window as admin and did it
there.

```
> cmake --install . --config DEBUG
```

After that you can run the test apps OK.

Notes about building lws
========================

You can download and install lws using the [vcpkg](https://github.com/Microsoft/vcpkg/) dependency manager:
```
git clone https://github.com/microsoft/vcpkg.git
cd vcpkg
./bootstrap-vcpkg.sh
./vcpkg integrate install
vcpkg install libwebsockets
```
The lws port in vcpkg is kept up to date by Microsoft team members and community contributors. If the version is out of date, please [create an issue or pull request](https://github.com/Microsoft/vcpkg/) on the vcpkg repository.

@section cm Introduction to CMake

CMake is a multi-platform build tool that can generate build files for many
different target platforms. See more info at http://www.cmake.org

CMake also allows/recommends you to do "out of source"-builds, that is,
the build files are separated from your sources, so there is no need to
create elaborate clean scripts to get a clean source tree, instead you
simply remove your build directory.

Libwebsockets has been tested to build successfully on the following platforms
with SSL support (for OpenSSL/wolfSSL/BoringSSL):

- Windows (Visual Studio)
- Windows (MinGW)
- Linux (x86 and ARM)
- OSX
- NetBSD


@section build1 Building the library and test apps

The project settings used by CMake to generate the platform specific build
files is called [CMakeLists.txt](../CMakeLists.txt). CMake then uses one of its "Generators" to
output a Visual Studio project or Make file for instance. To see a list of
the available generators for your platform, simply run the "cmake" command.

Note that by default OpenSSL will be linked, if you don't want SSL support
see below on how to toggle compile options.


@section bu Building on Unix:

1. Install CMake 2.8 or greater: http://cmake.org/cmake/resources/software.html
   (Most Unix distributions comes with a packaged version also)

2. Install OpenSSL.

3. Generate the build files (default is Make files):
```
        $ cd /path/to/src
        $ mkdir build
        $ cd build
        $ cmake ..
```

4. Finally you can build using the generated Makefile:
```
    $ make && sudo make install
```
**NOTE**: The `build/`` directory can have any name and be located anywhere
 on your filesystem, and that the argument `..` given to cmake is simply
 the source directory of **libwebsockets** containing the [CMakeLists.txt](../CMakeLists.txt)
 project file. All examples in this file assumes you use ".."

**NOTE2**:
A common option you may want to give is to set the install path, same
as --prefix= with autotools.  It defaults to /usr/local.
You can do this by, eg
```
    $ cmake -DCMAKE_INSTALL_PREFIX:PATH=/usr .
```

**NOTE3**:
On machines that want libraries in lib64, you can also add the
following to the cmake line
```
    -DLIB_SUFFIX=64
```

**NOTE4**:
If you are building against a non-distro OpenSSL (eg, in order to get
access to ALPN support only in newer OpenSSL versions) the nice way to
express that in one cmake command is eg,
```
    $ cmake .. -DOPENSSL_ROOT_DIR=/usr/local/ssl \
         -DCMAKE_INCLUDE_DIRECTORIES_PROJECT_BEFORE=/usr/local/ssl \
         -DLWS_WITH_HTTP2=1
```

When you run the test apps using non-distro SSL, you have to force them
to use your libs, not the distro ones
```
    $ LD_LIBRARY_PATH=/usr/local/ssl/lib libwebsockets-test-server --ssl
```

To get it to build on latest openssl (2016-04-10) it needed this approach
```
    cmake .. -DLWS_WITH_HTTP2=1 -DLWS_OPENSSL_INCLUDE_DIRS=/usr/local/include/openssl -DLWS_OPENSSL_LIBRARIES="/usr/local/lib64/libssl.so;/usr/local/lib64/libcrypto.so"
```

Mac users have reported

```
 $ export OPENSSL_ROOT_DIR=/usr/local/Cellar/openssl/1.0.2k; cmake ..; make -j4
```

worked for them when using "homebrew" OpenSSL

**NOTE5**:
To build with debug info and _DEBUG for lower priority debug messages
compiled in, use
```
    $ cmake .. -DCMAKE_BUILD_TYPE=DEBUG
```

**NOTE6**
To build on Solaris the linker needs to be informed to use lib socket
and libnsl, and only builds in 64bit mode.

```bash
    $ cmake .. -DCMAKE_C_FLAGS=-m64 -DCMAKE_EXE_LINKER_FLAGS="-lsocket -lnsl"
```

**NOTE7**

Build and test flow against boringssl.  Notice `LWS_WITH_GENHASH` is currently
unavailable with boringssl due to their removing the necessary apis.

Build current HEAD boringssl

```
 $ cd /projects
 $ git clone https://boringssl.googlesource.com/boringssl
 $ cd boringssl
 $ mkdir build
 $ cd build
 $ cmake ..  -DBUILD_SHARED_LIBS=1
 $ make -j8
```

Build and test lws against it

```
 $ cd /projects/libwebsockets/build
 $ cmake .. -DOPENSSL_LIBRARIES="/projects/boringssl/build/ssl/libssl.so;\
   /projects/boringssl/build/crypto/libcrypto.so" \
   -DOPENSSL_INCLUDE_DIRS=/projects/boringssl/include \
   -DLWS_WITH_BORINGSSL=1 -DCMAKE_BUILD_TYPE=DEBUG
 $ make -j8 && sudo make install
 $ LD_PRELOAD="/projects/boringssl/build/ssl/libssl.so \
   /projects/boringssl/build/crypto/libcrypto.so" \
   /usr/local/bin/libwebsockets-test-server -s
```

4. Finally you can build using the generated Makefile:

```bash
    $ make
 ```

@section lcap Linux Capabilities

On Linux, lws now lets you retain selected root capabilities when dropping
privileges.  If libcap-dev or similar package is installed providing
sys/capabilities.h, and libcap or similar package is installed providing
libcap.so, CMake will enable the capability features.

The context creation info struct .caps[] and .count_caps members can then
be set by user code to enable selected root capabilities to survive the
transition to running under an unprivileged user.

@section cmq Quirk of cmake

When changing cmake options, for some reason the only way to get it to see the
changes sometimes is delete the contents of your build directory and do the
cmake from scratch.

deleting build/CMakeCache.txt may be enough.


@section cmw Building on Windows (Visual Studio)

1. Install CMake 2.6 or greater: http://cmake.org/cmake/resources/software.html

2. Install OpenSSL binaries. https://wiki.openssl.org/index.php/Binaries

   (**NOTE**: Preferably in the default location to make it easier for CMake to find them)

   **NOTE2**:
   Be sure that OPENSSL_CONF environment variable is defined and points at
   <OpenSSL install location>\bin\openssl.cfg

3. Generate the Visual studio project by opening the Visual Studio cmd prompt:

```
    cd <path to src>
    md build
    cd build
    cmake -G "Visual Studio 10" ..
```

   (**NOTE**: There is also a cmake-gui available on Windows if you prefer that)

   **NOTE2**:
   See this link to find out the version number corresponding to your Visual Studio edition:
   http://superuser.com/a/194065

4. Now you should have a generated Visual Studio Solution in  your
   `<path to src>/build` directory, which can be used to build.

5. Some additional deps may be needed

 - iphlpapi.lib
 - psapi.lib
 - userenv.lib

6. If you're using libuv, you must make sure to compile libuv with the same multithread-dll / Mtd attributes as libwebsockets itself


@section cmwmgw Building on Windows (MinGW)

1. Install MinGW

    For Fedora, it's, eg, `dnf install mingw64-gcc`

2. Install current CMake package

    For Fedora, it's `dnf install cmake`

3. Instal mingw-built OpenSSL pieces

    For Fedora, it's `mingw64-openssl.noarch mingw64-openssl-static.noarch`

    mingw64-cmake as described below will auto-find the libs and includes
    for build.  But to execute the apps, they either need to go into the same
    `/usr/x86_64-w64-mingw32/sys-root/mingw/bin/` as the dlls are installed to,
    or the dlls have to be copied into the same dir as your app executable.

4. Generate the build files (default is Make files) using MSYS shell.

   For Fedora, they provide a `mingw64-cmake` wrapper in the package
   `mingw64-filesystem`, with this you can run that instead of cmake directly
   and don't have to get involved with setting the cmake generator.

   Otherwise doing it by hand is like this:

```
    $ cd /drive/path/to/src
    $ mkdir build
    $ cd build
    $ cmake -G "MSYS Makefiles" -DCMAKE_INSTALL_PREFIX=C:/MinGW ..
```

   To generate build files allowing to create libwebsockets binaries with debug information
   set the CMAKE_BUILD_TYPE flag to DEBUG:
```
    $ cmake -G "MSYS Makefiles" -DCMAKE_INSTALL_PREFIX=C:/MinGW -DCMAKE_BUILD_TYPE=DEBUG ..
```
5. Finally you can build using the generated Makefile and get the results deployed into your MinGW installation:

```
    $ make && make install
```

@section distro Selecting CMake options useful for distros

Distro packagers should select the CMake option "LWS_WITH_DISTRO_RECOMMENDED",
which selects common additional options like support for various event libraries,
plugins and lwsws.

@section ssllib Choosing Your TLS Poison

 - If you are really restricted on memory, code size, or don't care about TLS
   speed, mbedTLS is a good choice: `cmake .. -DLWS_WITH_MBEDTLS=1`

 - If cpu and memory is not super restricted and you care about TLS speed,
   OpenSSL or a directly compatible variant like Boring SSL is a good choice.

Just building lws against stock Fedora OpenSSL or stock Fedora mbedTLS, for
SSL handhake mbedTLS takes ~36ms and OpenSSL takes ~1ms on the same x86_64
build machine here, with everything else the same.  Over the 144 connections of
h2spec compliance testing for example, this ends up completing in 400ms for
OpenSSL and 5.5sec for mbedTLS on x86_64.  In other words mbedTLS is very slow
compared to OpenSSL under the (fairly typical) conditions I tested it.

This isn't an inefficiency in the mbedtls interface implementation, it's just
mbedTLS doing the crypto much slower than OpenSSL, which has accelerated
versions of common crypto operations it automatically uses for platforms
supporting it.  As of Oct 2017 mbedTLS itself has no such optimizations for any
platform that I could find.  It's just pure C running on the CPU.

Lws supports both almost the same, so instead of taking my word for it you are
invited to try it both ways and see which the results (including, eg, binary
size and memory usage as well as speed) suggest you use.

NOTE: one major difference with mbedTLS is it does not load the system trust
store by default.  That has advantages and disadvantages, but the disadvantage
is you must provide the CA cert to lws built against mbedTLS for it to be able
to validate it, ie, use -A with the test client.  The minimal test clients
have the CA cert for warmcat.com and libwebsockets.org and use it if they see
they were built with mbedTLS.

@section optee Building for OP-TEE

OP-TEE is a "Secure World" Trusted Execution Environment.

Although lws is only part of the necessary picture to have an https-enabled
TA, it does support OP-TEE as a platform and if you provide the other
pieces, does work very well.

Select it in cmake with `-DLWS_PLAT_OPTEE=1`


@section cmco Setting compile options

To set compile time flags you can either use one of the CMake gui applications
or do it via the command line.

@subsection cmcocl Command line

To list available options (omit the H if you don't want the help text):

    cmake -LH ..

Then to set an option and build (for example turn off SSL support):

    cmake -DLWS_WITH_SSL=0 ..
or
    cmake -DLWS_WITH_SSL:BOOL=OFF ..

@subsection cmcoug Unix GUI

If you have a curses-enabled build you simply type:
(not all packages include this, my debian install does not for example).

    ccmake

@subsection cmcowg Windows GUI

On windows CMake comes with a gui application:
    Start -> Programs -> CMake -> CMake (cmake-gui)


@section wolf wolfSSL/CyaSSL replacement for OpenSSL

wolfSSL/CyaSSL is a lightweight SSL library targeted at embedded systems:
https://www.wolfssl.com/wolfSSL/Products-wolfssl.html

It contains a OpenSSL compatibility layer which makes it possible to pretty
much link to it instead of OpenSSL, giving a much smaller footprint.

**NOTE**: wolfssl needs to be compiled using the `--enable-opensslextra` flag for
this to work.

@section wolf1 Compiling libwebsockets with wolfSSL

```
    cmake .. -DLWS_WITH_WOLFSSL=1 \
         -DLWS_WOLFSSL_INCLUDE_DIRS=/path/to/wolfssl \
         -DLWS_WOLFSSL_LIBRARIES=/path/to/wolfssl/wolfssl.a ..
```

**NOTE**: On windows use the .lib file extension for `LWS_WOLFSSL_LIBRARIES` instead.

@section cya Compiling libwebsockets with CyaSSL

```
    cmake .. -DLWS_WITH_CYASSL=1 \
         -DLWS_CYASSL_INCLUDE_DIRS=/path/to/cyassl \
         -DLWS_CYASSL_LIBRARIES=/path/to/wolfssl/cyassl.a ..
```

**NOTE**: On windows use the .lib file extension for `LWS_CYASSL_LIBRARIES` instead.

@section gzip Selecting GZIP or MINIZ

By default lws supports gzip when compression is needed.  But you can tell it to use
MINIZ instead by using `-DLWS_WITH_MINIZ=1`.

For native build cmake will try to find an existing libminiz.so or .a and build
against that and the found includes automatically.

For cross-build or building against local miniz, you need the following kind of
cmake to tell it where to get miniz

```
cmake .. -DLWS_WITH_MINIZ=1 -DLWS_WITH_ZIP_FOPS=1 -DMINIZ_INCLUDE_DIRS="/projects/miniz;/projects/miniz/build" -DMINIZ_LIBRARIES=/projects/miniz/build/libminiz.so.2.1.0
```

@section esp32 Building for ESP32

Building for ESP32 requires the ESP-IDF framework. It can be built under Linux, OSX or Windows (MSYS2).

1. Install ESP-IDF, follow the getting started guide here - http://esp-idf.readthedocs.io/en/latest/get-started/
2. Set ESP-IDF to last known working version (assuming ESP-IDF is in `~/esp/esp-idf`) :
```
    cd ~/esp/esp-idf
    git checkout 0c50b65a34cd6b3954f7435193411a88adb49cb0
    git submodule update --recursive
```
3. Add `libwebsockets` as a submodule in the `components` folder of your ESP-IDF project:
```
    git submodule add https://github.com/warmcat/libwebsockets.git components/libwebsockets
```
4. If on Windows (MSYS2) you will need to install CMake in the MSYS2 environment:
```
    pacman -S mingw-w64-i686-cmake
```
If you're on Linux or OSX ensure CMake version is at least 3.7.

@section extplugins Building plugins outside of lws itself

The directory ./plugin-standalone/ shows how easy it is to create plugins
outside of lws itself.  First build lws itself with -DLWS_WITH_PLUGINS,
then use the same flow to build the standalone plugin
```
    cd ./plugin-standalone
    mkdir build
    cd build
    cmake ..
    make && sudo make install
```

if you changed the default plugin directory when you built lws, you must
also give the same arguments to cmake here (eg,
` -DCMAKE_INSTALL_PREFIX:PATH=/usr/something/else...` )

Otherwise if you run lwsws or libwebsockets-test-server-v2.0, it will now
find the additional plugin "libprotocol_example_standalone.so"
```
    lwsts[21257]:   Plugins:
    lwsts[21257]:    libprotocol_dumb_increment.so
    lwsts[21257]:    libprotocol_example_standalone.so
    lwsts[21257]:    libprotocol_lws_mirror.so
    lwsts[21257]:    libprotocol_lws_server_status.so
    lwsts[21257]:    libprotocol_lws_status.so
```
If you have multiple vhosts, you must enable plugins at the vhost
additionally, discovered plugins are not enabled automatically for security
reasons.  You do this using info->pvo or for lwsws, in the JSON config.


@section http2rp Reproducing HTTP/2 tests

Enable `-DLWS_WITH_HTTP2=1` in cmake to build with http/2 support enabled.

You must have built and be running lws against a version of openssl that has
ALPN.  At the time of writing, recent distros have started upgrading to OpenSSL
1.1+ that supports this already.  You'll know it's right by seeing

```
    lwsts[4752]:  Compiled with OpenSSL support
    lwsts[4752]:  Using SSL mode
    lwsts[4752]:  HTTP2 / ALPN enabled
```
at lws startup.

Recent Firefox and Chrome also support HTTP/2 by ALPN, so these should just work
with the test server running in -s / ssl mode.

For testing with nghttp client:

```
    $ nghttp -nvas https://localhost:7681/test.html
```

Testing with h2spec (https://github.com/summerwind/h2spec)

```
        $ h2spec  -h 127.0.0.1 -p 7681 -t -k -v -o 1
```

```
145 tests, 145 passed, 0 skipped, 0 failed

```

@section coverage Automated Coverage Testing

./test-apps/attack.sh contains scripted tests that are the basis
of the automated test coverage assessment available for gcc and clang.

To reproduce

 $ cd build
 $ cmake .. -DLWS_WITH_GCOV=1 -DCMAKE_BUILD_TYPE=DEBUG
 $ ../scripts/build-gcov.sh
 $ ../test-apps/attack.sh
 $ ../scripts/gcov.sh
...
Lines executed:51.24% of 8279

@section windowsprebuilt Using Windows binary builds on Appveyor

The CI builds on Appveyor now produce usable binary outputs.  Visit

[lws on Appveyor](https://ci.appveyor.com/project/lws-team/libwebsockets)

and select one of the builds, then click on ARTIFACTS at the top right.  The zip file
want to be unpacked into `C:\Program Files (x86)/libwebsockets`, after that, you should be able to run the test server, by running it from `bin/Release/libwebsockets-test-server.exe` and opening a browser on http://127.0.0.1:7681

@section cross Cross compiling

To enable cross-compiling **libwebsockets** using CMake you need to create
a "Toolchain file" that you supply to CMake when generating your build files.
CMake will then use the cross compilers and build paths specified in this file
to look for dependencies and such.

**Libwebsockets** includes an example toolchain file [cross-arm-linux-gnueabihf.cmake](../contrib/cross-arm-linux-gnueabihf.cmake)
you can use as a starting point.

The commandline to configure for cross with this would look like
```
    $ cmake .. -DCMAKE_INSTALL_PREFIX:PATH=/usr/lib/my-cross-root \
         -DCMAKE_TOOLCHAIN_FILE=../contrib/cross-arm-linux-gnueabihf.cmake \
         -DLWS_WITHOUT_EXTENSIONS=1 -DLWS_WITH_SSL=0 \
         -DLWS_WITH_ZIP_FOPS=0 -DLWS_WITH_ZLIB=0
```
The example shows how to build with no external cross lib dependencies, you
need to provide the cross libraries otherwise.

**NOTE**: start from an EMPTY build directory if you had a non-cross build in there
    before the settings will be cached and your changes ignored.
    Delete `build/CMakeCache.txt` at least before trying a new cmake config
    to ensure you are really building the options you think you are.

Additional information on cross compilation with CMake:
    http://www.vtk.org/Wiki/CMake_Cross_Compiling

@section cross_example Complex Cross compiling example

Here are step by step instructions for cross-building the external projects needed for lws with lwsws + mbedtls as an example.

In the example, my toolchain lives in `/projects/aist-tb/arm-tc` and is named `arm-linux-gnueabihf`.  So you will need to adapt those to where your toolchain lives and its name where you see them here.

Likewise I do all this in /tmp but it has no special meaning, you can adapt that to somewhere else.

All "foreign" cross-built binaries are sent into `/tmp/cross` so they cannot be confused for 'native' x86_64 stuff on your host machine in /usr/[local/]....

## Prepare the cmake toolchain file

1) `cd /tmp`

2) `wget -O mytoolchainfile https://raw.githubusercontent.com/warmcat/libwebsockets/master/contrib/cross-arm-linux-gnueabihf.cmake`

3) Edit `/tmp/mytoolchainfile` adapting `CROSS_PATH`, `CMAKE_C_COMPILER` and `CMAKE_CXX_COMPILER` to reflect your toolchain install dir and path to your toolchain C and C++ compilers respectively.  For my case:

```
set(CROSS_PATH /projects/aist-tb/arm-tc/)
set(CMAKE_C_COMPILER "${CROSS_PATH}/bin/arm-linux-gnueabihf-gcc")
set(CMAKE_CXX_COMPILER "${CROSS_PATH}/bin/arm-linux-gnueabihf-g++")
```

## 1/4: Building libuv cross:

1) `export PATH=/projects/aist-tb/arm-tc/bin:$PATH`  Notice there is a **/bin** on the end of the toolchain path

2) `cd /tmp ; mkdir cross` we will put the cross-built libs in /tmp/cross

3) `git clone https://github.com/libuv/libuv.git` get libuv

4) `cd libuv`

5) `./autogen.sh`

```
+ libtoolize --copy
libtoolize: putting auxiliary files in '.'.
libtoolize: copying file './ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
libtoolize: copying file 'm4/libtool.m4'
libtoolize: copying file 'm4/ltoptions.m4'
libtoolize: copying file 'm4/ltsugar.m4'
libtoolize: copying file 'm4/ltversion.m4'
libtoolize: copying file 'm4/lt~obsolete.m4'
+ aclocal -I m4
+ autoconf
+ automake --add-missing --copy
configure.ac:38: installing './ar-lib'
configure.ac:25: installing './compile'
configure.ac:22: installing './config.guess'
configure.ac:22: installing './config.sub'
configure.ac:21: installing './install-sh'
configure.ac:21: installing './missing'
Makefile.am: installing './depcomp'
```
If it has problems, you will need to install `automake`, `libtool` etc.

6) `./configure  --host=arm-linux-gnueabihf --prefix=/tmp/cross`

7) `make && make install` this will install to `/tmp/cross/...`

8) `file /tmp/cross/lib/libuv.so.1.0.0`  Check it's really built for ARM
```
/tmp/cross/lib/libuv.so.1.0.0: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, BuildID[sha1]=cdde0bc945e51db6001a9485349c035baaec2b46, with debug_info, not stripped
```

## 2/4: Building zlib cross

1) `cd /tmp`

2) `git clone https://github.com/madler/zlib.git`

3) `CC=arm-linux-gnueabihf-gcc ./configure --prefix=/tmp/cross`
```
Checking for shared library support...
Building shared library libz.so.1.2.11 with arm-linux-gnueabihf-gcc.
Checking for size_t... Yes.
Checking for off64_t... Yes.
Checking for fseeko... Yes.
Checking for strerror... Yes.
Checking for unistd.h... Yes.
Checking for stdarg.h... Yes.
Checking whether to use vs[n]printf() or s[n]printf()... using vs[n]printf().
Checking for vsnprintf() in stdio.h... Yes.
Checking for return value of vsnprintf()... Yes.
Checking for attribute(visibility) support... Yes.
```

4)  `make && make install`
```
arm-linux-gnueabihf-gcc -O3 -D_LARGEFILE64_SOURCE=1 -DHAVE_HIDDEN -I. -c -o example.o test/example.c
...
rm -f /tmp/cross/include/zlib.h /tmp/cross/include/zconf.h
cp zlib.h zconf.h /tmp/cross/include
chmod 644 /tmp/cross/include/zlib.h /tmp/cross/include/zconf.h
```

5) `file /tmp/cross/lib/libz.so.1.2.11`  This is just to confirm we built an ARM lib as expected
```
/tmp/cross/lib/libz.so.1.2.11: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, BuildID[sha1]=6f8ffef84389b1417d2fd1da1bd0c90f748f300d, with debug_info, not stripped
```

## 3/4: Building mbedtls cross

1) `cd /tmp`

2) `git clone https://github.com/ARMmbed/mbedtls.git`

3) `cd mbedtls ; mkdir build ; cd build`

3) `cmake .. -DCMAKE_TOOLCHAIN_FILE=/tmp/mytoolchainfile -DCMAKE_INSTALL_PREFIX:PATH=/tmp/cross -DCMAKE_BUILD_TYPE=RELEASE -DUSE_SHARED_MBEDTLS_LIBRARY=1`  mbedtls also uses cmake, so you can simply reuse the toolchain file you used for libwebsockets.  That is why you shouldn't put project-specific options in the toolchain file, it should just describe the toolchain.

4) `make && make install`

5) `file /tmp/cross/lib/libmbedcrypto.so.2.6.0`
```
/tmp/cross/lib/libmbedcrypto.so.2.6.0: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, BuildID[sha1]=bcca195e78bd4fd2fb37f36ab7d72d477d609d87, with debug_info, not stripped
```

## 4/4: Building libwebsockets with everything

1) `cd /tmp`

2) `git clone ssh://git@github.com/warmcat/libwebsockets`

3) `cd libwebsockets ; mkdir build ; cd build`

4)  (this is all one line on the commandline)
```
cmake .. -DCMAKE_TOOLCHAIN_FILE=/tmp/mytoolchainfile \
-DCMAKE_INSTALL_PREFIX:PATH=/tmp/cross \
-DLWS_WITH_LWSWS=1 \
-DLWS_WITH_MBEDTLS=1 \
-DLWS_MBEDTLS_LIBRARIES="/tmp/cross/lib/libmbedcrypto.so;/tmp/cross/lib/libmbedtls.so;/tmp/cross/lib/libmbedx509.so" \
-DLWS_MBEDTLS_INCLUDE_DIRS=/tmp/cross/include \
-DLWS_LIBUV_LIBRARIES=/tmp/cross/lib/libuv.so \
-DLWS_LIBUV_INCLUDE_DIRS=/tmp/cross/include \
-DLWS_ZLIB_LIBRARIES=/tmp/cross/lib/libz.so \
-DLWS_ZLIB_INCLUDE_DIRS=/tmp/cross/include
```

3) `make && make install`

4) `file /tmp/cross/lib/libwebsockets.so.11`
```
/tmp/cross/lib/libwebsockets.so.11: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, BuildID[sha1]=81e59c6534f8e9629a9fc9065c6e955ce96ca690, with debug_info, not stripped
```

5) `arm-linux-gnueabihf-objdump -p /tmp/cross/lib/libwebsockets.so.11 | grep NEEDED`  Confirm that the lws library was linked against everything we expect (libm / libc are provided by your toolchain)
```
  NEEDED               libz.so.1
  NEEDED               libmbedcrypto.so.0
  NEEDED               libmbedtls.so.10
  NEEDED               libmbedx509.so.0
  NEEDED               libuv.so.1
  NEEDED               libm.so.6
  NEEDED               libc.so.6
```

You will also find the lws test apps in `/tmp/cross/bin`... to run lws on the target you will need to copy the related things from /tmp/cross... all the .so from /tmp/cross/lib and anything from /tmp/cross/bin you want.

@section mem Memory efficiency

Embedded server-only configuration without extensions (ie, no compression
on websocket connections), but with full v13 websocket features and http
server, built on ARM Cortex-A9:

Update at 8dac94d (2013-02-18)
```
    $ ./configure --without-client --without-extensions --disable-debug --without-daemonize

    Context Creation, 1024 fd limit[2]:   16720 (includes 12 bytes per fd)
    Per-connection [3]:                      72 bytes, +1328 during headers

    .text	.rodata	.data	.bss
    11512	2784	288	4
```
This shows the impact of the major configuration with/without options at
13ba5bbc633ea962d46d using Ubuntu ARM on a PandaBoard ES.

These are accounting for static allocations from the library elf, there are
additional dynamic allocations via malloc.  These are a bit old now but give
the right idea for relative "expense" of features.

Static allocations, ARM9

|                                | .text   | .rodata | .data | .bss |
|--------------------------------|---------|---------|-------|------|
| All (no without)               | 35024   | 9940    | 336   | 4104 |
| without client                 | 25684   | 7144    | 336   | 4104 |
| without client, exts           | 21652   | 6288    | 288   | 4104 |
| without client, exts, debug[1] | 19756   | 3768    | 288   | 4104 |
| without server                 | 30304   | 8160    | 336   | 4104 |
| without server, exts           | 25382   | 7204    | 288   | 4104 |
| without server, exts, debug[1] | 23712   | 4256    | 288   | 4104 |

[1] `--disable-debug` only removes messages below `lwsl_notice`.  Since that is
the default logging level the impact is not noticeable, error, warn and notice
logs are all still there.

[2] `1024` fd per process is the default limit (set by ulimit) in at least Fedora
and Ubuntu.  You can make significant savings tailoring this to actual expected
peak fds, ie, at a limit of `20`, context creation allocation reduces to `4432 +
240 = 4672`)

[3] known header content is freed after connection establishment

## Need for CI

Generally if we're adding something that's supposed to work ongoing, the stuff
should be exercised in CI (at least Travis).

If there are few users for a particular feature, experience has shown that
refactors or other upheaval can easily break it into a state of uselessness
without anyone noticing until later.

Therefore here's a description of how to add something to the CI tests... this
is certainly a nonproductive PITA and I have never been thanked for the work
involved.  But if the promise of the various features working is going to
remain alive, it's necessary to include CI test where possible with new
nontrivial code.

## Integration points

### cmake

`.travis.yml` maps the various test activities to CMake options needed.

### including dependent packages into travis

See `./scripts/travis_install.sh`

### performing prepared test actions

See `./scripts/travis_control.sh`

Notes about coding with lws
===========================

@section era Old lws and lws v2.0

Originally lws only supported the "manual" method of handling everything in the
user callback found in test-server.c / test-server-http.c.

Since v2.0, the need for most or all of this manual boilerplate has been
eliminated: the protocols[0] http stuff is provided by a generic lib export
`lws_callback_http_dummy()`.  You can serve parts of your filesystem at part of
the URL space using mounts, the dummy http callback will do the right thing.

It's much preferred to use the "automated" v2.0 type scheme, because it's less
code and it's easier to support.

The minimal examples all use the modern, recommended way.

If you just need generic serving capability, without the need to integrate lws
to some other app, consider not writing any server code at all, and instead use
the generic server `lwsws`, and writing your special user code in a standalone
"plugin".  The server is configured for mounts etc using JSON, see
./READMEs/README.lwsws.md.

Although the "plugins" are dynamically loaded if you use lwsws or lws built
with libuv, actually they may perfectly well be statically included if that
suits your situation better, eg, ESP32 test server, where the platform does
not support processes or dynamic loading, just #includes the plugins
one after the other and gets the same benefit from the same code.

Isolating and collating the protocol code in one place also makes it very easy
to maintain and understand.

So it if highly recommended you put your protocol-specific code into the
form of a "plugin" at the source level, even if you have no immediate plan to
use it dynamically-loaded.

@section writeable Only send data when socket writeable

You should only send data on a websocket connection from the user callback
`LWS_CALLBACK_SERVER_WRITEABLE` (or `LWS_CALLBACK_CLIENT_WRITEABLE` for
clients).

If you want to send something, do NOT just send it but request a callback
when the socket is writeable using

 - `lws_callback_on_writable(wsi)` for a specific `wsi`, or

 - `lws_callback_on_writable_all_protocol(protocol)` for all connections
using that protocol to get a callback when next writeable.

Usually you will get called back immediately next time around the service
loop, but if your peer is slow or temporarily inactive the callback will be
delayed accordingly.  Generating what to write and sending it should be done
in the ...WRITEABLE callback.

See the test server code for an example of how to do this.

Otherwise evolved libs like libuv get this wrong, they will allow you to "send"
anything you want but it only uses up your local memory (and costs you
memcpys) until the socket can actually accept it.  It is much better to regulate
your send action by the downstream peer readiness to take new data in the first
place, avoiding all the wasted buffering.

Libwebsockets' concept is that the downstream peer is truly the boss, if he,
or our connection to him, cannot handle anything new, we should not generate
anything new for him.  This is how unix shell piping works, you may have
`cat a.txt | grep xyz > remote", but actually that does not cat anything from
a.txt while remote cannot accept anything new.

@section oneper Only one lws_write per WRITEABLE callback

From v2.5, lws strictly enforces only one lws_write() per WRITEABLE callback.

You will receive a message about "Illegal back-to-back write of ... detected"
if there is a second lws_write() before returning to the event loop.

This is because with http/2, the state of the network connection carrying a
wsi is unrelated to any state of the wsi.  The situation on http/1 where a
new request implied a new tcp connection and new SSL buffer, so you could
assume some window for writes is no longer true.  Any lws_write() can fail
and be buffered for completion by lws; it will be auto-completed by the
event loop.

Note that if you are handling your own http responses, writing the headers
needs to be done with a separate lws_write() from writing any payload.  That
means after writing the headers you must call `lws_callback_on_writable(wsi)`
and send any payload from the writable callback.

@section otherwr Do not rely on only your own WRITEABLE requests appearing

Libwebsockets may generate additional `LWS_CALLBACK_CLIENT_WRITEABLE` events
if it met network conditions where it had to buffer your send data internally.

So your code for `LWS_CALLBACK_CLIENT_WRITEABLE` needs to own the decision
about what to send, it can't assume that just because the writeable callback
came something is ready to send.

It's quite possible you get an 'extra' writeable callback at any time and
just need to `return 0` and wait for the expected callback later.

@section dae Daemonization

There's a helper api `lws_daemonize` built by default that does everything you
need to daemonize well, including creating a lock file.  If you're making
what's basically a daemon, just call this early in your init to fork to a
headless background process and exit the starting process.

Notice stdout, stderr, stdin are all redirected to /dev/null to enforce your
daemon is headless, so you'll need to sort out alternative logging, by, eg,
syslog via `lws_set_log_level(..., lwsl_emit_syslog)`.

@section conns Maximum number of connections

The maximum number of connections the library can deal with is decided when
it starts by querying the OS to find out how many file descriptors it is
allowed to open (1024 on Fedora for example).  It then allocates arrays that
allow up to that many connections, minus whatever other file descriptors are
in use by the user code.

If you want to restrict that allocation, or increase it, you can use ulimit or
similar to change the available number of file descriptors, and when restarted
**libwebsockets** will adapt accordingly.

@section peer_limits optional LWS_WITH_PEER_LIMITS

If you select `LWS_WITH_PEER_LIMITS` at cmake, then lws will track peer IPs
and monitor how many connections and ah resources they are trying to use
at one time.  You can choose to limit these at context creation time, using
`info.ip_limit_ah` and `info.ip_limit_wsi`.

Note that although the ah limit is 'soft', ie, the connection will just wait
until the IP is under the ah limit again before attaching a new ah, the
wsi limit is 'hard', lws will drop any additional connections from the
IP until it's under the limit again.

If you use these limits, you should consider multiple clients may simultaneously
try to access the site through NAT, etc.  So the limits should err on the side
of being generous, while still making it impossible for one IP to exhaust
all the server resources.

@section evtloop Libwebsockets is singlethreaded

Libwebsockets works in a serialized event loop, in a single thread.  It supports
the default poll() backend, and libuv, libev, and libevent event loop
libraries that also take this locking-free, nonblocking event loop approach that
is not threadsafe.  There are several advantages to this technique, but one
disadvantage, it doesn't integrate easily if there are multiple threads that
want to use libwebsockets.

However integration to multithreaded apps is possible if you follow some guidelines.

1) Aside from two APIs, directly calling lws apis from other threads is not allowed.

2) If you want to keep a list of live wsi, you need to use lifecycle callbacks on
the protocol in the service thread to manage the list, with your own locking.
Typically you use an ESTABLISHED callback to add ws wsi to your list and a CLOSED
callback to remove them.

3) LWS regulates your write activity by being able to let you know when you may
write more on a connection.  That reflects the reality that you cannot succeed to
send data to a peer that has no room for it, so you should not generate or buffer
write data until you know the peer connection can take more.

Other libraries pretend that the guy doing the writing is the boss who decides
what happens, and absorb as much as you want to write to local buffering.  That does
not scale to a lot of connections, because it will exhaust your memory and waste
time copying data around in memory needlessly.

The truth is the receiver, along with the network between you, is the boss who
decides what will happen.  If he stops accepting data, no data will move.  LWS is
designed to reflect that.

If you have something to send, you call `lws_callback_on_writable()` on the
connection, and when it is writeable, you will get a `LWS_CALLBACK_SERVER_WRITEABLE`
callback, where you should generate the data to send and send it with `lws_write()`.

You cannot send data using `lws_write()` outside of the WRITEABLE callback.

4) For multithreaded apps, this corresponds to a need to be able to provoke the
`lws_callback_on_writable()` action and to wake the service thread from its event
loop wait (sleeping in `poll()` or `epoll()` or whatever).  The rules above
mean directly sending data on the connection from another thread is out of the
question.

Therefore the two apis mentioned above that may be used from another thread are

 - For LWS using the default poll() event loop, `lws_callback_on_writable()`

 - For LWS using libuv/libev/libevent event loop, `lws_cancel_service()`

If you are using the default poll() event loop, one "foreign thread" at a time may
call `lws_callback_on_writable()` directly for a wsi.  You need to use your own
locking around that to serialize multiple thread access to it.

If you implement LWS_CALLBACK_GET_THREAD_ID in protocols[0], then LWS will detect
when it has been called from a foreign thread and automatically use
`lws_cancel_service()` to additionally wake the service loop from its wait.

For libuv/libev/libevent event loop, they cannot handle being called from other
threads.  So there is a slightly different scheme, you may call `lws_cancel_service()`
to force the event loop to end immediately.  This then broadcasts a callback (in the
service thread context) `LWS_CALLBACK_EVENT_WAIT_CANCELLED`, to all protocols on all
vhosts, where you can perform your own locking and walk a list of wsi that need
`lws_callback_on_writable()` calling on them.

`lws_cancel_service()` is very cheap to call.

5) The obverse of this truism about the receiver being the boss is the case where
we are receiving.  If we get into a situation we actually can't usefully
receive any more, perhaps because we are passing the data on and the guy we want
to send to can't receive any more, then we should "turn off RX" by using the
RX flow control API, `lws_rx_flow_control(wsi, 0)`.  When something happens where we
can accept more RX, (eg, we learn our onward connection is writeable) we can call
it again to re-enable it on the incoming wsi.

LWS stops calling back about RX immediately you use flow control to disable RX, it
buffers the data internally if necessary.  So you will only see RX when you can
handle it.  When flow control is disabled, LWS stops taking new data in... this makes
the situation known to the sender by TCP "backpressure", the tx window fills and the
sender finds he cannot write any more to the connection.

See the mirror protocol implementations for example code.

If you need to service other socket or file descriptors as well as the
websocket ones, you can combine them together with the websocket ones
in one poll loop, see "External Polling Loop support" below, and
still do it all in one thread / process context.  If the need is less
architectural, you can also create RAW mode client and serving sockets; this
is how the lws plugin for the ssh server works.

@section anonprot Working without a protocol name

Websockets allows connections to negotiate without a protocol name...
in that case by default it will bind to the first protocol in your
vhost protocols[] array.

You can tell the vhost to use a different protocol by attaching a
pvo (per-vhost option) to the

```
/*
 * this sets a per-vhost, per-protocol option name:value pair
 * the effect is to set this protocol to be the default one for the vhost,
 * ie, selected if no Protocol: header is sent with the ws upgrade.
 */

static const struct lws_protocol_vhost_options pvo_opt = {
	NULL,
	NULL,
	"default",
	"1"
};

static const struct lws_protocol_vhost_options pvo = {
	NULL,
	&pvo_opt,
	"my-protocol",
	""
};

...

	context_info.pvo = &pvo;
...

```

Will select "my-protocol" from your protocol list (even if it came
in by plugin) as being the target of client connections that don't
specify a protocol.

@section closing Closing connections from the user side

When you want to close a connection, you do it by returning `-1` from a
callback for that connection.

You can provoke a callback by calling `lws_callback_on_writable` on
the wsi, then notice in the callback you want to close it and just return -1.
But usually, the decision to close is made in a callback already and returning
-1 is simple.

If the socket knows the connection is dead, because the peer closed or there
was an affirmitive network error like a FIN coming, then **libwebsockets**  will
take care of closing the connection automatically.

If you have a silently dead connection, it's possible to enter a state where
the send pipe on the connection is choked but no ack will ever come, so the
dead connection will never become writeable.  To cover that, you can use TCP
keepalives (see later in this document) or pings.

@section gzip Serving from inside a zip file

Lws now supports serving gzipped files from inside a zip container.  Thanks to
Per Bothner for contributing the code.

This has the advtantage that if the client can accept GZIP encoding, lws can
simply send the gzip-compressed file from inside the zip file with no further
processing, saving time and bandwidth.

In the case the client can't understand gzip compression, lws automatically
decompressed the file and sends it normally.

Clients with limited storage and RAM will find this useful; the memory needed
for the inflate case is constrained so that only one input buffer at a time
is ever in memory.

To use this feature, ensure LWS_WITH_ZIP_FOPS is enabled at CMake.

`libwebsockets-test-server-v2.0` includes a mount using this technology
already, run that test server and navigate to http://localhost:7681/ziptest/candide.html

This will serve the book Candide in html, together with two jpgs, all from
inside a .zip file in /usr/[local/]share-libwebsockets-test-server/candide.zip

Usage is otherwise automatic, if you arrange a mount that points to the zipfile,
eg, "/ziptest" -> "mypath/test.zip", then URLs like `/ziptest/index.html` will be
servied from `index.html` inside `mypath/test.zip`

@section frags Fragmented messages

To support fragmented messages you need to check for the final
frame of a message with `lws_is_final_fragment`. This
check can be combined with `libwebsockets_remaining_packet_payload`
to gather the whole contents of a message, eg:

```
	    case LWS_CALLBACK_RECEIVE:
	    {
	        Client * const client = (Client *)user;
	        const size_t remaining = lws_remaining_packet_payload(wsi);

	        if (!remaining && lws_is_final_fragment(wsi)) {
	            if (client->HasFragments()) {
	                client->AppendMessageFragment(in, len, 0);
	                in = (void *)client->GetMessage();
	                len = client->GetMessageLength();
	            }

	            client->ProcessMessage((char *)in, len, wsi);
	            client->ResetMessage();
	        } else
	            client->AppendMessageFragment(in, len, remaining);
	    }
	    break;
```

The test app libwebsockets-test-fraggle sources also show how to
deal with fragmented messages.


@section debuglog Debug Logging

Also using `lws_set_log_level` api you may provide a custom callback to actually
emit the log string.  By default, this points to an internal emit function
that sends to stderr.  Setting it to `NULL` leaves it as it is instead.

A helper function `lwsl_emit_syslog()` is exported from the library to simplify
logging to syslog.  You still need to use `setlogmask`, `openlog` and `closelog`
in your user code.

The logging apis are made available for user code.

- `lwsl_err(...)`
- `lwsl_warn(...)`
- `lwsl_notice(...)`
- `lwsl_info(...)`
- `lwsl_debug(...)`

The difference between notice and info is that notice will be logged by default
whereas info is ignored by default.

If you are not building with _DEBUG defined, ie, without this

```
	$ cmake .. -DCMAKE_BUILD_TYPE=DEBUG
```

then log levels below notice do not actually get compiled in.

@section asan Building with ASAN

Under GCC you can select for the build to be instrumented with the Address
Sanitizer, using `cmake .. -DCMAKE_BUILD_TYPE=DEBUG -DLWS_WITH_ASAN=1`.  LWS is routinely run during development with valgrind, but ASAN is capable of finding different issues at runtime, like operations which are not strictly defined in the C
standard and depend on platform behaviours.

Run your application like this

```
	$ sudo ASAN_OPTIONS=verbosity=2:halt_on_error=1  /usr/local/bin/lwsws
```

and attach gdb to catch the place it halts.

@section extpoll External Polling Loop support

**libwebsockets** maintains an internal `poll()` array for all of its
sockets, but you can instead integrate the sockets into an
external polling array.  That's needed if **libwebsockets** will
cooperate with an existing poll array maintained by another
server.

Three callbacks `LWS_CALLBACK_ADD_POLL_FD`, `LWS_CALLBACK_DEL_POLL_FD`
and `LWS_CALLBACK_CHANGE_MODE_POLL_FD` appear in the callback for protocol 0
and allow interface code to manage socket descriptors in other poll loops.

You can pass all pollfds that need service to `lws_service_fd()`, even
if the socket or file does not belong to **libwebsockets** it is safe.

If **libwebsocket** handled it, it zeros the pollfd `revents` field before returning.
So you can let **libwebsockets** try and if `pollfd->revents` is nonzero on return,
you know it needs handling by your code.

Also note that when integrating a foreign event loop like libev or libuv where
it doesn't natively use poll() semantics, and you must return a fake pollfd
reflecting the real event:

 - be sure you set .events to .revents value as well in the synthesized pollfd

 - check the built-in support for the event loop if possible (eg, ./lib/libuv.c)
   to see how it interfaces to lws

 - use LWS_POLLHUP / LWS_POLLIN / LWS_POLLOUT from libwebsockets.h to avoid
   losing windows compatibility

You also need to take care about "forced service" somehow... these are cases
where the network event was consumed, incoming data was all read, for example,
but the work arising from it was not completed.  There will not be any more
network event to trigger the remaining work, Eg, we read compressed data, but
we did not use up all the decompressed data before returning to the event loop
because we had to write some of it.

Lws provides an API to determine if anyone is waiting for forced service,
`lws_service_adjust_timeout(context, 1, tsi)`, normally tsi is 0.  If it returns
0, then at least one connection has pending work you can get done by calling
`lws_service_tsi(context, -1, tsi)`, again normally tsi is 0.

For eg, the default poll() event loop, or libuv/ev/event, lws does this
checking for you and handles it automatically.  But in the external polling
loop case, you must do it explicitly.  Handling it after every normal service
triggered by the external poll fd should be enough, since the situations needing
it are initially triggered by actual network events.

An example of handling it is shown in the test-server code specific to
external polling.

@section cpp Using with in c++ apps

The library is ready for use by C++ apps.  You can get started quickly by
copying the test server

```
	$ cp test-apps/test-server.c test.cpp
```

and building it in C++ like this

```
	$ g++ -DINSTALL_DATADIR=\"/usr/share\" -ocpptest test.cpp -lwebsockets
```

`INSTALL_DATADIR` is only needed because the test server uses it as shipped, if
you remove the references to it in your app you don't need to define it on
the g++ line either.


@section headerinfo Availability of header information

HTTP Header information is managed by a pool of "ah" structs.  These are a
limited resource so there is pressure to free the headers and return the ah to
the pool for reuse.

For that reason header information on HTTP connections that get upgraded to
websockets is lost after the ESTABLISHED callback.  Anything important that
isn't processed by user code before then should be copied out for later.

For HTTP connections that don't upgrade, header info remains available the
whole time.

@section http2compat Code Requirements for HTTP/2 compatibility

Websocket connections only work over http/1, so there is nothing special to do
when you want to enable -DLWS_WITH_HTTP2=1.

The internal http apis already follow these requirements and are compatible with
http/2 already.  So if you use stuff like mounts and serve stuff out of the
filesystem, there's also nothing special to do.

However if you are getting your hands dirty with writing response headers, or
writing bulk data over http/2, you need to observe these rules so that it will
work over both http/1.x and http/2 the same.

1) LWS_PRE requirement applies on ALL lws_write().  For http/1, you don't have
to take care of LWS_PRE for http data, since it is just sent straight out.
For http/2, it will write up to LWS_PRE bytes behind the buffer start to create
the http/2 frame header.

This has implications if you treated the input buffer to lws_write() as const...
it isn't any more with http/2, up to 9 bytes behind the buffer will be trashed.

2) Headers are encoded using a sophisticated scheme in http/2.  The existing
header access apis are already made compatible for incoming headers,
for outgoing headers you must:

 - observe the LWS_PRE buffer requirement mentioned above

 - Use `lws_add_http_header_status()` to add the transaction status (200 etc)

 - use lws apis `lws_add_http_header_by_name()` and `lws_add_http_header_by_token()`
   to put the headers into the buffer (these will translate what is actually
   written to the buffer depending on if the connection is in http/2 mode or not)

 - use the `lws api lws_finalize_http_header()` api after adding the last
   response header

 - write the header using lws_write(..., `LWS_WRITE_HTTP_HEADERS`);

 3) http/2 introduces per-stream transmit credit... how much more you can send
 on a stream is decided by the peer.  You start off with some amount, as the
 stream sends stuff lws will reduce your credit accordingly, when it reaches
 zero, you must not send anything further until lws receives "more credit" for
 that stream the peer.  Lws will suppress writable callbacks if you hit 0 until
 more credit for the stream appears, and lws built-in file serving (via mounts
 etc) already takes care of observing the tx credit restrictions.  However if
 you write your own code that wants to send http data, you must consult the
 `lws_get_peer_write_allowance()` api to find out the state of your tx credit.
 For http/1, it will always return (size_t)-1, ie, no limit.

 This is orthogonal to the question of how much space your local side's kernel
 will make to buffer your send data on that connection.  So although the result
 from `lws_get_peer_write_allowance()` is "how much you can send" logically,
 and may be megabytes if the peer allows it, you should restrict what you send
 at one time to whatever your machine will generally accept in one go, and
 further reduce that amount if `lws_get_peer_write_allowance()` returns
 something smaller.  If it returns 0, you should not consume or send anything
 and return having asked for callback on writable, it will only come back when
 more tx credit has arrived for your stream.

 4) Header names with captital letters are illegal in http/2.  Header names in
 http/1 are case insensitive.  So if you generate headers by name, change all
 your header name strings to lower-case to be compatible both ways.

 5) Chunked Transfer-encoding is illegal in http/2, http/2 peers will actively
 reject it.  Lws takes care of removing the header and converting CGIs that
 emit chunked into unchunked automatically for http/2 connections.

If you follow these rules, your code will automatically work with both http/1.x
and http/2.

@section ka TCP Keepalive

It is possible for a connection which is not being used to send to die
silently somewhere between the peer and the side not sending.  In this case
by default TCP will just not report anything and you will never get any more
incoming data or sign the link is dead until you try to send.

To deal with getting a notification of that situation, you can choose to
enable TCP keepalives on all **libwebsockets** sockets, when you create the
context.

To enable keepalive, set the ka_time member of the context creation parameter
struct to a nonzero value (in seconds) at context creation time.  You should
also fill ka_probes and ka_interval in that case.

With keepalive enabled, the TCP layer will send control packets that should
stimulate a response from the peer without affecting link traffic.  If the
response is not coming, the socket will announce an error at `poll()` forcing
a close.

Note that BSDs don't support keepalive time / probes / interval per-socket
like Linux does.  On those systems you can enable keepalive by a nonzero
value in `ka_time`, but the systemwide kernel settings for the time / probes/
interval are used, regardless of what nonzero value is in `ka_time`.


@section sslopt Optimizing SSL connections

There's a member `ssl_cipher_list` in the `lws_context_creation_info` struct
which allows the user code to restrict the possible cipher selection at
context-creation time.

You might want to look into that to stop the ssl peers selecting a cipher which
is too computationally expensive.  To use it, point it to a string like

	`"RC4-MD5:RC4-SHA:AES128-SHA:AES256-SHA:HIGH:!DSS:!aNULL"`

if left `NULL`, then the "DEFAULT" set of ciphers are all possible to select.

You can also set it to `"ALL"` to allow everything (including insecure ciphers).


@section sslcerts Passing your own cert information direct to SSL_CTX

For most users it's enough to pass the SSL certificate and key information by
giving filepaths to the info.ssl_cert_filepath and info.ssl_private_key_filepath
members when creating the vhost.

If you want to control that from your own code instead, you can do so by leaving
the related info members NULL, and setting the info.options flag
LWS_SERVER_OPTION_CREATE_VHOST_SSL_CTX at vhost creation time.  That will create
the vhost SSL_CTX without any certificate, and allow you to use the callback
LWS_CALLBACK_OPENSSL_LOAD_EXTRA_SERVER_VERIFY_CERTS to add your certificate to
the SSL_CTX directly.  The vhost SSL_CTX * is in the user parameter in that
callback.

@section clientasync Async nature of client connections

When you call `lws_client_connect_info(..)` and get a `wsi` back, it does not
mean your connection is active.  It just means it started trying to connect.

Your client connection is actually active only when you receive
`LWS_CALLBACK_CLIENT_ESTABLISHED` for it.

There's a 5 second timeout for the connection, and it may give up or die for
other reasons, if any of that happens you'll get a
`LWS_CALLBACK_CLIENT_CONNECTION_ERROR` callback on protocol 0 instead for the
`wsi`.

After attempting the connection and getting back a non-`NULL` `wsi` you should
loop calling `lws_service()` until one of the above callbacks occurs.

As usual, see [test-client.c](../test-apps/test-client.c) for example code.

Notice that the client connection api tries to progress the connection
somewhat before returning.  That means it's possible to get callbacks like
CONNECTION_ERROR on the new connection before your user code had a chance to
get the wsi returned to identify it (in fact if the connection did fail early,
NULL will be returned instead of the wsi anyway).

To avoid that problem, you can fill in `pwsi` in the client connection info
struct to point to a struct lws that get filled in early by the client
connection api with the related wsi.  You can then check for that in the
callback to confirm the identity of the failing client connection.


@section fileapi Lws platform-independent file access apis

lws now exposes his internal platform file abstraction in a way that can be
both used by user code to make it platform-agnostic, and be overridden or
subclassed by user code.  This allows things like handling the URI "directory
space" as a virtual filesystem that may or may not be backed by a regular
filesystem.  One example use is serving files from inside large compressed
archive storage without having to unpack anything except the file being
requested.

The test server shows how to use it, basically the platform-specific part of
lws prepares a file operations structure that lives in the lws context.

The user code can get a pointer to the file operations struct

```
	LWS_VISIBLE LWS_EXTERN struct lws_plat_file_ops *
		`lws_get_fops`(struct lws_context *context);
```

and then can use helpers to also leverage these platform-independent
file handling apis

```
	lws_fop_fd_t
	`lws_plat_file_open`(struct lws_plat_file_ops *fops, const char *filename,
			   lws_fop_flags_t *flags)
	int
	`lws_plat_file_close`(lws_fop_fd_t fop_fd)

	unsigned long
	`lws_plat_file_seek_cur`(lws_fop_fd_t fop_fd, lws_fileofs_t offset)

	int
	`lws_plat_file_read`(lws_fop_fd_t fop_fd, lws_filepos_t *amount,
		   uint8_t *buf, lws_filepos_t len)

	int
	`lws_plat_file_write`(lws_fop_fd_t fop_fd, lws_filepos_t *amount,
		   uint8_t *buf, lws_filepos_t len )
```

Generic helpers are provided which provide access to generic fops information or
call through to the above fops

```
lws_filepos_t
lws_vfs_tell(lws_fop_fd_t fop_fd);

lws_filepos_t
lws_vfs_get_length(lws_fop_fd_t fop_fd);

uint32_t
lws_vfs_get_mod_time(lws_fop_fd_t fop_fd);

lws_fileofs_t
lws_vfs_file_seek_set(lws_fop_fd_t fop_fd, lws_fileofs_t offset);

lws_fileofs_t
lws_vfs_file_seek_end(lws_fop_fd_t fop_fd, lws_fileofs_t offset);
```


The user code can also override or subclass the file operations, to either
wrap or replace them.  An example is shown in test server.

### Changes from v2.1 and before fops

There are several changes:

1) Pre-2.2 fops directly used platform file descriptors.  Current fops returns and accepts a wrapper type lws_fop_fd_t which is a pointer to a malloc'd struct containing information specific to the filesystem implementation.

2) Pre-2.2 fops bound the fops to a wsi.  This is completely removed, you just give a pointer to the fops struct that applies to this file when you open it.  Afterwards, the operations in the fops just need the lws_fop_fd_t returned from the open.

3) Everything is wrapped in typedefs.  See lws-plat-unix.c for examples of how to implement.

4) Position in the file, File Length, and a copy of Flags left after open are now generically held in the fop_fd.
VFS implementation must set and manage this generic information now.  See the implementations in lws-plat-unix.c for
examples.

5) The file length is no longer set at a pointer provided by the open() fop.  The api `lws_vfs_get_length()` is provided to
get the file length after open.

6) If your file namespace is virtual, ie, is not reachable by platform fops directly, you must set LWS_FOP_FLAG_VIRTUAL
on the flags during open.

7) There is an optional `mod_time` uint32_t member in the generic fop_fd.  If you are able to set it during open, you
should indicate it by setting `LWS_FOP_FLAG_MOD_TIME_VALID` on the flags.

@section rawfd RAW file descriptor polling

LWS allows you to include generic platform file descriptors in the lws service / poll / event loop.

Open your fd normally and then

```
	lws_sock_file_fd_type u;

	u.filefd = your_open_file_fd;

	if (!lws_adopt_descriptor_vhost(vhost, 0, u,
					"protocol-name-to-bind-to",
					optional_wsi_parent_or_NULL)) {
		// failed
	}

	// OK
```

A wsi is created for the file fd that acts like other wsi, you will get these
callbacks on the named protocol

```
	LWS_CALLBACK_RAW_ADOPT_FILE
	LWS_CALLBACK_RAW_RX_FILE
	LWS_CALLBACK_RAW_WRITEABLE_FILE
	LWS_CALLBACK_RAW_CLOSE_FILE
```

starting with LWS_CALLBACK_RAW_ADOPT_FILE.

The minimal example `raw/minimal-raw-file` demonstrates how to use it.

`protocol-lws-raw-test` plugin also provides a method for testing this with
`libwebsockets-test-server-v2.0`:

The plugin creates a FIFO on your system called "/tmp/lws-test-raw"

You can feed it data through the FIFO like this

```
  $ sudo sh -c "echo hello > /tmp/lws-test-raw"
```

This plugin simply prints the data.  But it does it through the lws event
loop / service poll.

@section rawsrvsocket RAW server socket descriptor polling

You can also enable your vhost to accept RAW socket connections, in addition to
HTTP[s] and WS[s].  If the first bytes written on the connection are not a
valid HTTP method, then the connection switches to RAW mode.

This is disabled by default, you enable it by setting the `.options` flag
LWS_SERVER_OPTION_FALLBACK_TO_APPLY_LISTEN_ACCEPT_CONFIG, and setting
`.listen_accept_role` to `"raw-skt"` when creating the vhost.

RAW mode socket connections receive the following callbacks

```
	LWS_CALLBACK_RAW_ADOPT
	LWS_CALLBACK_RAW_RX
	LWS_CALLBACK_RAW_WRITEABLE
	LWS_CALLBACK_RAW_CLOSE
```

You can control which protocol on your vhost handles these RAW mode
incoming connections by setting the vhost info struct's `.listen_accept_protocol`
to the vhost protocol name to use.

`protocol-lws-raw-test` plugin provides a method for testing this with
`libwebsockets-test-server-v2.0`:

Run libwebsockets-test-server-v2.0 and connect to it by telnet, eg

```
    $ telnet 127.0.0.1 7681
```

type something that isn't a valid HTTP method and enter, before the
connection times out.  The connection will switch to RAW mode using this
protocol, and pass the unused rx as a raw RX callback.

The test protocol echos back what was typed on telnet to telnet.

@section rawclientsocket RAW client socket descriptor polling

You can now also open RAW socket connections in client mode.

Follow the usual method for creating a client connection, but set the
`info.method` to "RAW".  When the connection is made, the wsi will be
converted to RAW mode and operate using the same callbacks as the
server RAW sockets described above.

The libwebsockets-test-client supports this using raw:// URLS.  To
test, open a netcat listener in one window

```
 $ nc -l 9999
```

and in another window, connect to it using the test client

```
 $ libwebsockets-test-client raw://127.0.0.1:9999
```

The connection should succeed, and text typed in the netcat window (including a CRLF)
will be received in the client.

@section rawudp RAW UDP socket integration

Lws provides an api to create, optionally bind, and adopt a RAW UDP
socket (RAW here means an uninterpreted normal UDP socket, not a
"raw socket").

```
LWS_VISIBLE LWS_EXTERN struct lws *
lws_create_adopt_udp(struct lws_vhost *vhost, int port, int flags,
		     const char *protocol_name, struct lws *parent_wsi);
```

`flags` should be `LWS_CAUDP_BIND` if the socket will receive packets.

The callbacks `LWS_CALLBACK_RAW_ADOPT`, `LWS_CALLBACK_RAW_CLOSE`,
`LWS_CALLBACK_RAW_RX` and `LWS_CALLBACK_RAW_WRITEABLE` apply to the
wsi.  But UDP is different than TCP in some fundamental ways.

For receiving on a UDP connection, data becomes available at
`LWS_CALLBACK_RAW_RX` as usual, but because there is no specific
connection with UDP, it is necessary to also get the source address of
the data separately, using `struct lws_udp * lws_get_udp(wsi)`.
You should take a copy of the `struct lws_udp` itself (not the
pointer) and save it for when you want to write back to that peer.

Writing is also a bit different for UDP.  By default, the system has no
idea about the receiver state and so asking for a `callback_on_writable()`
always believes that the socket is writeable... the callback will
happen next time around the event loop.

With UDP, there is no single "connection".  You need to write with sendto() and
direct the packets to a specific destination.  To return packets to a
peer who sent something earlier and you copied his `struct lws_udp`, you
use the .sa and .salen members as the last two parameters of the sendto().

The kernel may not accept to buffer / write everything you wanted to send.
So you are responsible to watch the result of sendto() and resend the
unsent part next time (which may involve adding new protocol headers to
the remainder depending on what you are doing).

@section ecdh ECDH Support

ECDH Certs are now supported.  Enable the CMake option

	cmake .. -DLWS_SSL_SERVER_WITH_ECDH_CERT=1

**and** the info->options flag

	LWS_SERVER_OPTION_SSL_ECDH

to build in support and select it at runtime.

@section sslinfo SSL info callbacks

OpenSSL allows you to receive callbacks for various events defined in a
bitmask in openssl/ssl.h.  The events include stuff like TLS Alerts.

By default, lws doesn't register for these callbacks.

However if you set the info.ssl_info_event_mask to nonzero (ie, set some
of the bits in it like `SSL_CB_ALERT` at vhost creation time, then
connections to that vhost will call back using LWS_CALLBACK_SSL_INFO
for the wsi, and the `in` parameter will be pointing to a struct of
related args:

```
struct lws_ssl_info {
	int where;
	int ret;
};
```

The default callback handler in lws has a handler for LWS_CALLBACK_SSL_INFO
which prints the related information,  You can test it using the switch
-S -s  on `libwebsockets-test-server-v2.0`.

Returning nonzero from the callback will close the wsi.

@section smp SMP / Multithreaded service

SMP support is integrated into LWS without any internal threading.  It's
very simple to use, libwebsockets-test-server-pthread shows how to do it,
use -j n argument there to control the number of service threads up to 32.

Two new members are added to the info struct

	unsigned int count_threads;
	unsigned int fd_limit_per_thread;

leave them at the default 0 to get the normal singlethreaded service loop.

Set count_threads to n to tell lws you will have n simultaneous service threads
operating on the context.

There is still a single listen socket on one port, no matter how many
service threads.

When a connection is made, it is accepted by the service thread with the least
connections active to perform load balancing.

The user code is responsible for spawning n threads running the service loop
associated to a specific tsi (Thread Service Index, 0 .. n - 1).  See
the libwebsockets-test-server-pthread for how to do.

If you leave fd_limit_per_thread at 0, then the process limit of fds is shared
between the service threads; if you process was allowed 1024 fds overall then
each thread is limited to 1024 / n.

You can set fd_limit_per_thread to a nonzero number to control this manually, eg
the overall supported fd limit is less than the process allowance.

You can control the context basic data allocation for multithreading from Cmake
using -DLWS_MAX_SMP=, if not given it's set to 1.  The serv_buf allocation
for the threads (currently 4096) is made at runtime only for active threads.

Because lws will limit the requested number of actual threads supported
according to LWS_MAX_SMP, there is an api lws_get_count_threads(context) to
discover how many threads were actually allowed when the context was created.

See the test-server-pthreads.c sample for how to use.

@section smplocking SMP Locking Helpers

Lws provide a set of pthread mutex helpers that reduce to no code or
variable footprint in the case that LWS_MAX_SMP == 1.

Define your user mutex like this

```
	lws_pthread_mutex(name);
```

If LWS_MAX_SMP > 1, this produces `pthread_mutex_t name;`.  In the case
LWS_MAX_SMP == 1, it produces nothing.

Likewise these helpers for init, destroy, lock and unlock


```
	void lws_pthread_mutex_init(pthread_mutex_t *lock)
	void lws_pthread_mutex_destroy(pthread_mutex_t *lock)
	void lws_pthread_mutex_lock(pthread_mutex_t *lock)
	void lws_pthread_mutex_unlock(pthread_mutex_t *lock)
```

resolve to nothing if LWS_MAX_SMP == 1, otherwise produce the equivalent
pthread api.

pthreads is required in lws only if LWS_MAX_SMP > 1.


@section libevuv libev / libuv / libevent support

You can select either or both

	-DLWS_WITH_LIBEV=1
	-DLWS_WITH_LIBUV=1
	-DLWS_WITH_LIBEVENT=1

at cmake configure-time.  The user application may use one of the
context init options flags

	LWS_SERVER_OPTION_LIBEV
	LWS_SERVER_OPTION_LIBUV
	LWS_SERVER_OPTION_LIBEVENT

to indicate it will use one of the event libraries at runtime.

libev and libevent headers conflict, they both define critical constants like
EV_READ to different values.  Attempts to discuss clearing that up with both
libevent and libev did not get anywhere useful.  Therefore CMakeLists.txt will
error out if you enable both LWS_WITH_LIBEV and LWS_WITH_LIBEVENT.

In addition depending on libev / compiler version, building anything with libev
apis using gcc may blow strict alias warnings (which are elevated to errors in
lws).  I did some googling at found these threads related to it, the issue goes
back at least to 2010 on and off

https://github.com/redis/hiredis/issues/434
https://bugs.gentoo.org/show_bug.cgi?id=615532
http://lists.schmorp.de/pipermail/libev/2010q1/000916.html
http://lists.schmorp.de/pipermail/libev/2010q1/000920.html
http://lists.schmorp.de/pipermail/libev/2010q1/000923.html

We worked around this problem by disabling -Werror on the parts of lws that
use libev.  FWIW as of Dec 2019 using Fedora 31 libev 4.27.1 and its gcc 9.2.1
doesn't seem to trigger the problem even without the workaround.

For these reasons and the response I got trying to raise these issues with
them, if you have a choice about event loop, I would gently encourage you
to avoid libev.  Where lws uses an event loop itself, eg in lwsws, we use
libuv.

@section extopts Extension option control from user code

User code may set per-connection extension options now, using a new api
`lws_set_extension_option()`.

This should be called from the ESTABLISHED callback like this
```
	 lws_set_extension_option(wsi, "permessage-deflate",
	                          "rx_buf_size", "12"); /* 1 << 12 */
```

If the extension is not active (missing or not negotiated for the
connection, or extensions are disabled on the library) the call is
just returns -1.  Otherwise the connection's extension has its
named option changed.

The extension may decide to alter or disallow the change, in the
example above permessage-deflate restricts the size of his rx
output buffer also considering the protocol's rx_buf_size member.


@section httpsclient Client connections as HTTP[S] rather than WS[S]

You may open a generic http client connection using the same
struct lws_client_connect_info used to create client ws[s]
connections.

To stay in http[s], set the optional info member "method" to
point to the string "GET" instead of the default NULL.

After the server headers are processed, when payload from the
server is available the callback LWS_CALLBACK_RECEIVE_CLIENT_HTTP
will be made.

You can choose whether to process the data immediately, or
queue a callback when an outgoing socket is writeable to provide
flow control, and process the data in the writable callback.

Either way you use the api `lws_http_client_read()` to access the
data, eg

```
	case LWS_CALLBACK_RECEIVE_CLIENT_HTTP:
		{
			char buffer[1024 + LWS_PRE];
			char *px = buffer + LWS_PRE;
			int lenx = sizeof(buffer) - LWS_PRE;

			lwsl_notice("LWS_CALLBACK_RECEIVE_CLIENT_HTTP\n");

			/*
			 * Often you need to flow control this by something
			 * else being writable.  In that case call the api
			 * to get a callback when writable here, and do the
			 * pending client read in the writeable callback of
			 * the output.
			 */
			if (lws_http_client_read(wsi, &px, &lenx) < 0)
				return -1;
			while (lenx--)
				putchar(*px++);
		}
		break;
```

Notice that if you will use SSL client connections on a vhost, you must
prepare the client SSL context for the vhost after creating the vhost, since
this is not normally done if the vhost was set up to listen / serve.  Call
the api lws_init_vhost_client_ssl() to also allow client SSL on the vhost.

@section clipipe Pipelining Client Requests to same host

If you are opening more client requests to the same host and port, you
can give the flag LCCSCF_PIPELINE on `info.ssl_connection` to indicate
you wish to pipeline them.

Without the flag, the client connections will occur concurrently using a
socket and tls wrapper if requested for each connection individually.
That is fast, but resource-intensive.

With the flag, lws will queue subsequent client connections on the first
connection to the same host and port.  When it has confirmed from the
first connection that pipelining / keep-alive is supported by the server,
it lets the queued client pipeline connections send their headers ahead
of time to create a pipeline of requests on the server side.

In this way only one tcp connection and tls wrapper is required to transfer
all the transactions sequentially.  It takes a little longer but it
can make a significant difference to resources on both sides.

If lws learns from the first response header that keepalive is not possible,
then it marks itself with that information and detaches any queued clients
to make their own individual connections as a fallback.

Lws can also intelligently combine multiple ongoing client connections to
the same host and port into a single http/2 connection with multiple
streams if the server supports it.

Unlike http/1 pipelining, with http/2 the client connections all occur
simultaneously using h2 stream multiplexing inside the one tcp + tls
connection.

You can turn off the h2 client support either by not building lws with
`-DLWS_WITH_HTTP2=1` or giving the `LCCSCF_NOT_H2` flag in the client
connection info struct `ssl_connection` member.

@section vhosts Using lws vhosts

If you set LWS_SERVER_OPTION_EXPLICIT_VHOSTS options flag when you create
your context, it won't create a default vhost using the info struct
members for compatibility.  Instead you can call lws_create_vhost()
afterwards to attach one or more vhosts manually.

```
	LWS_VISIBLE struct lws_vhost *
	lws_create_vhost(struct lws_context *context,
			 struct lws_context_creation_info *info);
```

lws_create_vhost() uses the same info struct as lws_create_context(),
it ignores members related to context and uses the ones meaningful
for vhost (marked with VH in libwebsockets.h).

```
	struct lws_context_creation_info {
		int port;					/* VH */
		const char *iface;				/* VH */
		const struct lws_protocols *protocols;		/* VH */
		const struct lws_extension *extensions;		/* VH */
	...
```

When you attach the vhost, if the vhost's port already has a listen socket
then both vhosts share it and use SNI (is SSL in use) or the Host: header
from the client to select the right one.  Or if no other vhost already
listening the a new listen socket is created.

There are some new members but mainly it's stuff you used to set at
context creation time.


@section sni How lws matches hostname or SNI to a vhost

LWS first strips any trailing :port number.

Then it tries to find an exact name match for a vhost listening on the correct
port, ie, if SNI or the Host: header provided abc.com:1234, it will match on a
vhost named abc.com that is listening on port 1234.

If there is no exact match, lws will consider wildcard matches, for example
if cats.abc.com:1234 is provided by the client by SNI or Host: header, it will
accept a vhost "abc.com" listening on port 1234.  If there was a better, exact,
match, it will have been chosen in preference to this.

Connections with SSL will still have the client go on to check the
certificate allows wildcards and error out if not.



@section mounts Using lws mounts on a vhost

The last argument to lws_create_vhost() lets you associate a linked
list of lws_http_mount structures with that vhost's URL 'namespace', in
a similar way that unix lets you mount filesystems into areas of your /
filesystem how you like and deal with the contents transparently.

```
	struct lws_http_mount {
		struct lws_http_mount *mount_next;
		const char *mountpoint; /* mountpoint in http pathspace, eg, "/" */
		const char *origin; /* path to be mounted, eg, "/var/www/warmcat.com" */
		const char *def; /* default target, eg, "index.html" */

		struct lws_protocol_vhost_options *cgienv;

		int cgi_timeout;
		int cache_max_age;

		unsigned int cache_reusable:1;
		unsigned int cache_revalidate:1;
		unsigned int cache_intermediaries:1;

		unsigned char origin_protocol;
		unsigned char mountpoint_len;
	};
```

The last mount structure should have a NULL mount_next, otherwise it should
point to the 'next' mount structure in your list.

Both the mount structures and the strings must persist until the context is
destroyed, since they are not copied but used in place.

`.origin_protocol` should be one of

```
	enum {
		LWSMPRO_HTTP,
		LWSMPRO_HTTPS,
		LWSMPRO_FILE,
		LWSMPRO_CGI,
		LWSMPRO_REDIR_HTTP,
		LWSMPRO_REDIR_HTTPS,
		LWSMPRO_CALLBACK,
	};
```

 - LWSMPRO_FILE is used for mapping url namespace to a filesystem directory and
serve it automatically.

 - LWSMPRO_CGI associates the url namespace with the given CGI executable, which
runs when the URL is accessed and the output provided to the client.

 - LWSMPRO_REDIR_HTTP and LWSMPRO_REDIR_HTTPS auto-redirect clients to the given
origin URL.

 - LWSMPRO_CALLBACK causes the http connection to attach to the callback
associated with the named protocol (which may be a plugin).


@section mountcallback Operation of LWSMPRO_CALLBACK mounts

The feature provided by CALLBACK type mounts is binding a part of the URL
namespace to a named protocol callback handler.

This allows protocol plugins to handle areas of the URL namespace.  For example
in test-server-v2.0.c, the URL area "/formtest" is associated with the plugin
providing "protocol-post-demo" like this

```
	static const struct lws_http_mount mount_post = {
		NULL,		/* linked-list pointer to next*/
		"/formtest",		/* mountpoint in URL namespace on this vhost */
		"protocol-post-demo",	/* handler */
		NULL,	/* default filename if none given */
		NULL,
		0,
		0,
		0,
		0,
		0,
		LWSMPRO_CALLBACK,	/* origin points to a callback */
		9,			/* strlen("/formtest"), ie length of the mountpoint */
	};
```

Client access to /formtest[anything] will be passed to the callback registered
with the named protocol, which in this case is provided by a protocol plugin.

Access by all methods, eg, GET and POST are handled by the callback.

protocol-post-demo deals with accepting and responding to the html form that
is in the test server HTML.

When a connection accesses a URL related to a CALLBACK type mount, the
connection protocol is changed until the next access on the connection to a
URL outside the same CALLBACK mount area.  User space on the connection is
arranged to be the size of the new protocol user space allocation as given in
the protocol struct.

This allocation is only deleted / replaced when the connection accesses a
URL region with a different protocol (or the default protocols[0] if no
CALLBACK area matches it).

This "binding connection to a protocol" lifecycle in managed by
`LWS_CALLBACK_HTTP_BIND_PROTOCOL` and `LWS_CALLBACK_HTTP_DROP_PROTOCOL`.
Because of HTTP/1.1 connection pipelining, one connection may perform
many transactions, each of which may map to different URLs and need
binding to different protocols.  So these messages are used to
create the binding of the wsi to your protocol including any
allocations, and to destroy the binding, at which point you should
destroy any related allocations.

@section BINDTODEV SO_BIND_TO_DEVICE

The .bind_iface flag in the context / vhost creation struct lets you
declare that you want all traffic for listen and transport on that
vhost to be strictly bound to the network interface named in .iface.

This Linux-only feature requires SO_BIND_TO_DEVICE, which in turn
requires CAP_NET_RAW capability... root has this capability.

However this feature needs to apply the binding also to accepted
sockets during normal operation, which implies the server must run
the whole time as root.

You can avoid this by using the Linux capabilities feature to have
the unprivileged user inherit just the CAP_NET_RAW capability.

You can confirm this with the test server


```
 $ sudo /usr/local/bin/libwebsockets-test-server -u agreen -i eno1 -k
```

The part that ensures the capability is inherited by the unprivileged
user is

```
#if defined(LWS_HAVE_SYS_CAPABILITY_H) && defined(LWS_HAVE_LIBCAP)
                        info.caps[0] = CAP_NET_RAW;
                        info.count_caps = 1;
#endif
```


@section dim Dimming webpage when connection lost

The lws test plugins' html provides useful feedback on the webpage about if it
is still connected to the server, by greying out the page if not.  You can
also add this to your own html easily

 - include lws-common.js from your HEAD section

   \<script src="/lws-common.js">\</script>

 - dim the page during initialization, in a script section on your page

   lws_gray_out(true,{'zindex':'499'});

 - in your ws onOpen(), remove the dimming

   lws_gray_out(false);

 - in your ws onClose(), reapply the dimming

   lws_gray_out(true,{'zindex':'499'});

@section errstyle Styling http error pages

In the code, http errors should be handled by `lws_return_http_status()`.

There are basically two ways... the vhost can be told to redirect to an "error
page" URL in response to specifically a 404... this is controlled by the
context / vhost info struct (`struct lws_context_creation_info`) member
`.error_document_404`... if non-null the client is redirected to this string.

If it wasn't redirected, then the response code html is synthesized containing
the user-selected text message and attempts to pull in `/error.css` for styling.

If this file exists, it can be used to style the error page.  See
https://libwebsockets.org/git/badrepo for an example of what can be done (
and https://libwebsockets.org/error.css for the corresponding css).

## Using Content Security Policy (CSP)

### What is it?

Modern browsers have recently implemented a new feature providing
a sort of "selinux for your web page".  If the server sends some
new headers describing the security policy for the content, then
the browser strictly enforces it.

### Why would we want to do that?

Scripting on webpages is pretty universal, sometimes the scripts
come from third parties, and sometimes attackers find a way to
inject scripts into the DOM, eg, through scripts in content.

CSP lets the origin server define what is legitimate for the page it
served and everything else is denied.

The CSP for warmcat.com and libwebsockets.org looks like this,
I removed a handful of whitelisted image sources like travis
status etc for clarity...

```
"content-security-policy": "default-src 'none'; img-src 'self' data:; script-src 'self'; font-src 'self'; style-src 'self'; connect-src 'self'; frame-ancestors 'none'; base-uri 'none';",
"x-content-type-options": "nosniff",
"x-xss-protection": "1; mode=block",
"x-frame-options": "deny",
"referrer-policy": "no-referrer"
```

The result of this is the browser won't let the site content be iframed, and it
will reject any inline styles or inline scripts.  Fonts, css, ajax, ws and
images are only allowed to come from 'self', ie, the server that served the
page.  You may inject your script, or deceptive styles: it won't run or be shown.

Because inline scripts are banned, the usual methods for XSS are dead;
the attacker can't even load js from another server.  So these rules
provide a very significant increase in client security.

### Implications of strict CSP

Halfhearted CSP isn't worth much.  The only useful approach is to start
with `default-src 'none'` which disables everything, and then whitelist the
minimum needed for the pages to operate.

"Minimum needed for the pages to operate" doesn't mean defeat the protections
necessary so everything in the HTML can stay the same... it means adapt the
pages to want the minimum and then enable the minimum.

The main point is segregation of styles and script away from the content, in
files referenced in the document `<head>` section, along these lines:

```
<head>
 <meta charset=utf-8 http-equiv="Content-Language" content="en"/>
 <link rel="stylesheet" type="text/css" href="test.css"/>
 <script type='text/javascript' src="/lws-common.js"></script>
 <script type='text/javascript' src='test.js'></script>
 <title>Minimal Websocket test app</title>
</head>
```

#### Inline styles must die

All styling must go in one or more `.css` file(s) best served by the same
server... while you can whitelist other sources in the CSP if you have to,
unless you control that server as well, you are allowing whoever gains
access to that server access to your users.

Inline styles are no longer allowed (eg, "style='font-size:120%'" in the
HTML)... they must be replaced by reference to one or more CSS class, which
in this case includes "font-size:120%".  This has always been the best
practice anyway, and your pages will be cleaner and more maintainable.

#### Inline scripts must die

Inline scripts need to be placed in a `.js` file and loaded in the page head
section, again it should only be from the server that provided the page.

Then, any kind of inline script, yours or injected or whatever, will be
completely rejected by the browser.

#### onXXX must be replaced by eventListener

Inline `onclick()` etc are kinds of inline scripting and are banned.

Modern browsers have offered a different system called ["EventListener" for
a while](https://developer.mozilla.org/en-US/docs/Web/API/EventListener)
which allows binding of events to DOM elements in JS.

A bunch of different named events are possible to listen on, commonly the
`.js` file will ask for one or both of

```
window.addEventListener("load", function() {
...
}, false);

document.addEventListener("DOMContentLoaded", function() {
...
}, false);
```

These give the JS a way to trigger when either everything on the page has
been "loaded" or the DOM has been populated from the initial HTML.  These
can set up other event listeners on the DOM objects and aftwards the
events will drive what happens on the page from user interaction and / or
timers etc.

If you have `onclick` in your HTML today, you would replace it with an id
for the HTML element, then eg in the DOMContentLoaded event listener,
apply

```
   document.getElementById("my-id").addEventListener("click", function() {
   ...
   }, false);
```

ie the .js file becomes the only place with the "business logic" of the
elements mentioned in the HTML, applied at runtime.

#### Do you really need external sources?

Do your scripts and fonts really need to come from external sources?
If your caching policy is liberal, they are not actually that expensive
to serve once and then the user is using his local copy for the next
days.

Some external sources are marked as anti-privacy in modern browsers, meaning
they track your users, in turn meaning if your site refers to them, you
will lose your green padlock in the browser.  If the content license allows
it, hosting them on "self", ie, the same server that provided the HTML,
will remove that problem.

Bringing in scripts from external sources is actually quite scary from the
security perspective.  If someone hacks the `ajax.googleapis.com` site to serve
a hostile, modified jquery, half the Internet will instantly
become malicious.  However if you serve it yourself, unless your server
was specifically targeted you know it will continue to serve what you
expect.

Since these scripts are usually sent with cache control headers for local
caching duration of 1 year, the cost of serving them yourself under the same
conditions is small but your susceptibility to attack is reduced to only taking
care of your own server.  And there is a privacy benefit that google is not
informed of your users' IPs and activities on your site.

## Contributing to lws

### How to contribute

Sending a patch with a bug report is very welcome.

For nontrivial problems, it's probably best to discuss on the mailing list,
or on github if you prefer, how to best solve it.

However your contribution is coming is fine:

 - paste a `git diff`

 - send a patch series by mail or mailing list

 - paste in a github issue

 - github PR

are all OK.

### Coding Standards

Code should look roughly like the existing code, which follows linux kernel
coding style.

If there are non-functional problems I will clean them out when I apply the
patch.

If there are functional problems (eg broken error paths etc) if they are
small compared to the working part I will also clean them.  If there are
larger problems, or consequences to the patch will have to discuss how to
solve them with a retry.

### Funding specific work

If there is a feature you wish was supported in lws, consider paying for the
work to be done.  The maintainer is a consultant and if we can agree the
task, you can quickly get a high quality result that does just what you need,
maintained ongoing along with the rest of lws.

# Lws Crypto Apis

## Overview

![lws crypto overview](/doc-assets/lws-crypto-overview.svg)

Lws provides a "generic" crypto layer on top of both OpenSSL and
compatible tls library, and mbedtls.  Using this layer, your code
can work without any changes on both types of tls library crypto
backends... it's as simple as rebuilding lws with `-DLWS_WITH_MBEDTLS=0`
or `=1` at cmake.

The generic layer can be used directly (as in, eg, the sshd plugin),
or via another layer on top, which processes JOSE JSON objects using
JWS (JSON Web Signatures), JWK (JSON Web Keys), and JWE (JSON Web
Encryption).

The `JW` apis use the generic apis (`lws_genrsa_`, etc) to get the crypto tasks
done, so anything they can do you can also get done using the generic apis.
The main difference is that with the generic apis, you must instantiate the
correct types and use type-specfic apis.  With the `JW` apis, there is only
one interface for all operations, with the details hidden in the api and
controlled by the JSON objects.

Because of this, the `JW` apis are often preferred because they give you
"crypto agility" cheaply... to change your crypto to another supported algorithm
once it's working, you literally just change your JSON defining the keys and
JWE or JWS algorithm.  (It's up to you to define your policy for which
combinations are acceptable by querying the parsed JW structs).

## Crypto supported in generic layer

### Generic Hash

 - SHA1
 - SHA256
 - SHA384
 - SHA512

### Generic HMAC

 - SHA256
 - SHA384
 - SHA512

### Generic AES

 - CBC
 - CFB128
 - CFB8
 - CTR
 - ECB
 - OFB
 - XTS
 - GCM
 - KW (Key Wrap)

### Generic RSA

 - PKCS 1.5
 - OAEP / PSS

### Generic EC

 - ECDH
 - ECDSA
 - P256 / P384 / P521 (sic) curves

## Using the generic layer

All the necessary includes are part of `libwebsockets.h`.

Enable `-DLWS_WITH_GENCRYPTO=1` at cmake.

|api|header|Functionality|
|---|---|---|
|genhash|[./include/libwebsockets/lws-genhash.h](https://libwebsockets.org/git/libwebsockets/tree/include/libwebsockets/lws-genhash.h)|Provides SHA1 + SHA2 hashes and hmac|
|genrsa|[./include/libwebsockets/lws-genrsa.h](https://libwebsockets.org/git/libwebsockets/tree/include/libwebsockets/lws-genrsa.h)|Provides RSA encryption, decryption, signing, verification, key generation and creation|
|genaes|[./include/libwebsockets/lws-genaes.h](https://libwebsockets.org/git/libwebsockets/tree/include/libwebsockets/lws-genaes.h)|Provides AES in all common variants for encryption and decryption|
|genec|[./include/libwebsockets/lws-genec.h](https://libwebsockets.org/git/libwebsockets/tree/include/libwebsockets/lws-genec.h)|Provides Elliptic Curve for encryption, decryption, signing, verification, key generation and creation|
|x509|[./include/libwebsockets/lws-x509.h](https://libwebsockets.org/git/libwebsockets/tree/include/libwebsockets/lws-x509.h)|Apis for X.509 Certificate loading, parsing, and stack verification, plus JWK key extraction from PEM X.509 certificate / private key|

Unit tests for these apis, which serve as usage examples, can be found in [./minimal-examples/api-tests/api-test-gencrypto](https://libwebsockets.org/git/libwebsockets/tree/minimal-examples/api-tests/api-test-gencrypto)

### Keys in the generic layer

The necessary types and defines are brought in by `libwebsockets.h`.

Keys are represented only by an array of `struct lws_jwk_elements`... the
length of the array is defined by the cipher... it's one of

|key elements count|definition|
|---|---|
|`LWS_COUNT_OCT_KEY_ELEMENTS`|1|
|`LWS_COUNT_RSA_KEY_ELEMENTS`|8|
|`LWS_COUNT_EC_KEY_ELEMENTS`|4|
|`LWS_COUNT_AES_KEY_ELEMENTS`|1|

`struct lws_jwk_elements` is a simple pointer / length combination used to
store arbitrary octets that make up the key element's binary representation.

## Using the JOSE layer

The JOSE (JWK / JWS / JWE) stuff is a crypto-agile JSON-based layer
that uses the gencrypto support underneath.

"Crypto Agility" means the JSON structs include information about the
algorithms and ciphers used in that particular object, making it easy to
upgrade system crypto strength or cycle keys over time while supporting a
transitional period where the old and new keys or algorithms + ciphers
are also valid.

Uniquely lws generic support means the JOSE stuff also has "tls library
agility", code written to the lws generic or JOSE apis is completely unchanged
even if the underlying tls library changes between OpenSSL and mbedtls, meaning
sharing code between server and client sides is painless.

All the necessary includes are part of `libwebsockets.h`.

Enable `-DLWS_WITH_JOSE=1` at CMake.

|api|header|Functionality|
|---|---|---|
|JOSE|[./include/libwebsockets/lws-jose.h](https://libwebsockets.org/git/libwebsockets/tree/include/libwebsockets/lws-jose.h)|Provides crypto agility for JWS / JWE|
|JWE|[./include/libwebsockets/lws-jwe.h](https://libwebsockets.org/git/libwebsockets/tree/include/libwebsockets/lws-jwe.h)|Provides Encryption and Decryption services for RFC7516 JWE JSON|
|JWS|[./include/libwebsockets/lws-jws.h](https://libwebsockets.org/git/libwebsockets/tree/include/libwebsockets/lws-jws.h)|Provides signature and verifcation services for RFC7515 JWS JSON|
|JWK|[./include/libwebsockets/lws-jwk.h](https://libwebsockets.org/git/libwebsockets/tree/include/libwebsockets/lws-jwk.h)|Provides signature and verifcation services for RFC7517 JWK JSON, both "keys" arrays and singletons|

Minimal examples are provided in the form of commandline tools for JWK / JWS / JWE / x509 handling:

 - [JWK minimal example](https://libwebsockets.org/git/libwebsockets/tree/minimal-examples/crypto/minimal-crypto-jwk)
 - [JWS minimal example](https://libwebsockets.org/git/libwebsockets/tree/minimal-examples/crypto/minimal-crypto-jws)
 - [JWE minimal example](https://libwebsockets.org/git/libwebsockets/tree/minimal-examples/crypto/minimal-crypto-jwe)
 - [X509 minimal example](https://libwebsockets.org/git/libwebsockets/tree/minimal-examples/crypto/minimal-crypto-x509)

Unit tests for these apis, which serve as usage examples, can be found in [./minimal-examples/api-tests/api-test-jose](https://libwebsockets.org/git/libwebsockets/tree/minimal-examples/api-tests/api-test-jose)

## Crypto supported in the JOSE layer

The JOSE RFCs define specific short names for different algorithms

### JWS

|JSOE name|Hash|Signature|
---|---|---
|RS256, RS384, RS512|SHA256/384/512|RSA
|ES256, ES384, ES521|SHA256/384/512|EC

### JWE

|Key Encryption|Payload authentication + crypt|
|---|---|
|`RSAES-PKCS1-v1.5` 2048b & 4096b|`AES_128_CBC_HMAC_SHA_256`|
|`RSAES-PKCS1-v1.5` 2048b|`AES_192_CBC_HMAC_SHA_384`|
|`RSAES-PKCS1-v1.5` 2048b|`AES_256_CBC_HMAC_SHA_512`|
|`RSAES-OAEP`|`AES_256_GCM`|
|`AES128KW`, `AES192KW`, `AES256KW`|`AES_128_CBC_HMAC_SHA_256`|
|`AES128KW`, `AES192KW`, `AES256KW`|`AES_192_CBC_HMAC_SHA_384`|
|`AES128KW`, `AES192KW`, `AES256KW`|`AES_256_CBC_HMAC_SHA_512`|
|`ECDH-ES` (P-256/384/521 key)|`AES_128/192/256_GCM`|
|`ECDH-ES+A128/192/256KW` (P-256/384/521 key)|`AES_128/192/256_GCM`|

### Keys in the JOSE layer

Keys in the JOSE layer use a `struct lws_jwk`, this contains two arrays of
`struct lws_jwk_elements` sized for the worst case (currently RSA).  One
array contains the key elements as described for the generic case, and the
other contains various nonencrypted key metadata taken from JWK JSON.

|metadata index|function|
|---|---|
|`JWK_META_KTY`|Key type, eg, "EC"|
|`JWK_META_KID`|Arbitrary ID string|
|`JWK_META_USE`|What the public key may be used to validate, "enc" or "sig"|
|`JWK_META_KEY_OPS`|Which operations the key is authorized for, eg, "encrypt"|
|`JWK_META_X5C`|Optional X.509 cert version of the key|
|`JWK_META_ALG`|Optional overall crypto algorithm the key is intended for use with|

`lws_jwk_destroy()` should be called when the jwk is going out of scope... this
takes care to zero down any key element data in the jwk.

# lws detailed latency

![lws detailed latency example plot](../doc-assets/lws-detailed-latency-example.png)

## Introduction

lws has the capability to make detailed latency measurements and
report them in realtime to a specified callback.

A default callback is provided that renders the data as text in
space-separated format suitable for gnuplot, to a specified file.

## Configuring

Enable `LWS_WITH_DETAILED_LATENCY` at cmake.

Create your context with something similar to this

```
#if defined(LWS_WITH_DETAILED_LATENCY)
	info.detailed_latency_cb = lws_det_lat_plot_cb;
	info.detailed_latency_filepath = "/tmp/lws-latency-results";
#endif
```

`lws_det_lat_plot_cb` is provided by lws as a convenience to convert
the stuct data provided at the callback interface to space-separated
text data that is easy to process with shell commands and gnuplot.

## `lws_det_lat_plot_cb` format

```
728239173547 N 23062 0 0 23062 0 0 0
728239192554 C 18879 0 0 18879 0 0 0
728239217894 T 25309 0 0 25309 0 0 0
728239234998 r 0 0 0 0 271 172 256
728239250611 r 0 0 0 0 69 934 4096
728239255679 w 19 122 18 159 20 80 80
728239275718 w 20 117 15 152 18 80 80
728239295578 w 10 73 7 90 7 80 80
728239315567 w 9 67 5 81 7 80 80
728239335745 w 23 133 9 165 14 80 80
...
```

Each event is shown in 9 columns

 - unix time in us
 - event type
   - N = Name resolution
   - C = TCP Connection
   - T = TLS negotiation server
   - t = TLS negotiation client
   - r = Read
   - w = Write
 - us duration, for w time client spent waiting to write
 - us duration, for w time data spent in transit to proxy
 - us duration, for w time proxy waited to send data
 - as a convenience, sum of last 3 columns above
 - us duration, time spent in callback
 - last 2 are actual / requested size in bytes

## Processing captured data with ministat

Eg, to summarize overall latencies on all captured writes

```
 $ cat /tmp/lws-latency-results | grep " w " | cut -d' ' -f6 | ministat
...
    N           Min           Max        Median           Avg        Stddev
x 1000            43           273           141       132.672     32.471693
```

## Processing captured data with gnuplot

### Gnuplot plotting script

Create a gnuplot script, eg myscript.gp

```
reset
set term pngcairo enhanced nocrop font "OpenSans, 12" size 800,600#output terminal and file
set output "lws-latency.png"
#set yrange [0:10000]
#to put an empty boundary around the
#data inside an autoscaled graph.
set offset graph 0.05,0.05,0.05,0.0
set style fill transparent solid 0.5 #fillstyle
set tics out nomirror
set xlabel "event"
set ylabel "latency (us)"
set format x ""
set title "Write latency"
set key invert reverse Right inside nobox
set key autotitle columnheader
set style data histogram
set style histogram rowstacked
set style fill solid border -1
set boxwidth 0.75
set style fill solid 1.00 noborder
set tic scale 0
set grid ytics lc rgb "#505050"
unset border
unset xtics

plot '/tmp/1' \
	   using ($3 + $4 + $5):xtic(1)         w boxes lt rgbcolor "blue"  title 'prox wr wait', \
	'' using ($3 + $4):xtic(1)         w boxes lt rgbcolor "green" title 'txfr to prox', \
	'' using 3:xtic(1) w boxes lt rgbcolor "red"   title 'cli wri wait'
```

### gnuplot invocation

```
 $ cat /tmp/lws-latency-results | grep " w " \>/tmp/1 ; gnuplot myscript.gp && eog lws-latency.png
```

ESP32 Support
=============

See \ref esp32 for details on how to build lws as a component in an ESP-IDF project.

Lws provides a "factory" application

https://github.com/warmcat/lws-esp32-factory

and a test application which implements the generic lws server test apps

https://github.com/warmcat/lws-esp32-test-server-demos

The behaviours of the generic factory are are quite rich, and cover uploading SSL certs through factory and user configuration, AP selection and passphrase entry, and managing a switch to allow the user to force entry to user setup mode at boot subsequently.

The factory app comes with partitioning for a 1MB factory partition containing that app and data, and a single 2.9MB OTA partition containing the main app.

The factory app is able to do OTA updates for both the factory and OTA partition slots; updating the factory slot first writes the new image to the OTA slot and copies it into place at the next boot, after which the user can reload the OTA slot.

State|Image|AP SSID|Port|URL|Mode
---|---|---|---|---|---
Factory Reset or Uninitialized|Factory|AP: ESP_012345|80|http://192.168.4.1|factory.html - to set certificates and serial
User configuration|Factory|AP: config-model-serial|443|https://192.168.4.1|index.html - user set up his AP information
Operation|OTA|Station only|443|https://model-serial.local|OTA application

## Basic Auth

The lws-esp32-test-server-demos app also demos basic auth.

On a normal platform this is done by binding a mount to a text file somewhere in the filesystem, which
contains user:password information one per line.

On ESP32 there is not necessarily any generic VFS in use.  So instead, the basic auth lookup is bound to
a given nvs domain, where the username is the key and the password the value.  main/main.c in the test
demos app shows how to both make the mount use basic auth, and how to set a user:password combination
using nvs.

Notes about generic-sessions Plugin
===================================

@section gseb Enabling lwsgs for build

Enable at CMake with -DLWS_WITH_GENERIC_SESSIONS=1

This also needs sqlite3 (libsqlite3-dev or similar package)


@section gsi lwsgs Introduction

The generic-sessions protocol plugin provides cookie-based login
authentication for lws web and ws connections.

The plugin handles everything about generic account registration,
email verification, lost password, account deletion, and other generic account
management.

Other code, in another eg, ws protocol handler, only needs very high-level
state information from generic-sessions, ie, which user the client is
authenticated as.  Everything underneath is managed in generic-sessions.


 - random 20-byte session id managed in a cookie

 - all information related to the session held at the server, nothing managed clientside

 - sqlite3 used at the server to manage active sessions and users

 - defaults to creating anonymous sessions with no user associated

 - admin account (with user-selectable username) is defined in config with a SHA-1 of the password; rest of the accounts are in sqlite3

 - user account passwords stored as salted SHA-1 with additional confounder
  only stored in the JSON config, not the database

 - login, logout, register account + email verification built-in with examples

 - in a mount, some file suffixes (ie, .js) can be associated with a protocol for the purposes of rewriting symbolnames.  These are read-only copies of logged-in server state.

 - When your page fetches .js or other rewritten files from that mount, "$lwsgs_user" and so on are rewritten on the fly using chunked transfer encoding

 - Eliminates server-side scripting with a few rewritten symbols and
 javascript on client side

 - 32-bit bitfield for authentication sectoring, mounts can provide a mask on the loggin-in session's associated server-side bitfield that must be set for access.

 - No code (just config) required for, eg, private URL namespace that requires login to access.


@section gsin Lwsgs Integration to HTML

Only three steps are needed to integrate lwsgs in your HTML.

1) lwsgs HTML UI is bundled with the javascript it uses in `lwsgs.js`, so
import that script file in your head section

2) define an empty div of id "lwsgs" somewhere

3) Call lwsgs_initial() in your page

That's it.  An example is below

```
	<html>
	 <head>
	  <script src="lwsgs.js"></script>
	  <style>
	     .body { font-size: 12 }
	     .gstitle { font-size: 18 }
	  </style>
	  </head>
	  <body style="background-image:url(seats.jpg)">
	    <table style="width:100%;transition: max-height 2s;">
	     <tr>
	      <td style="vertical-align:top;text-align:left;width=200px">
	       <img src="lwsgs-logo.png">
	      </td>
	      <td style="vertical-align:top;float:right">
		<div id=lwsgs style="text-align:right;background-color: rgba(255, 255, 255, 0.8);"></div>
	      </td>
	     </tr>
	    </table>
	   </form>

	   <script>lwsgs_initial();</script>

	 </body>
	</html>
```

@section gsof Lwsgs Overall Flow@

When the protocol is initialized, it gets per-vhost information from the config, such
as where the sqlite3 databases are to be stored.  The admin username and sha-1 of the
admin password are also taken from here.

In the mounts using protocol-generic-sessions, a cookie is maintained against any requests; if no cookie was active on the initial request a new session is
created with no attached user.

So there should always be an active session after any transactions with the server.

In the example html going to the mount /lwsgs loads a login / register page as the default.

The <form> in the login page contains 'next url' hidden inputs that let the html 'program' where the form handler will go after a successful admin login, a successful user login and a failed login.

After a successful login, the sqlite record at the server for the current session is updated to have the logged-in username associated with it.



@section gsconf Lwsgs Configuration

"auth-mask" defines the authorization sector bits that must be enabled on the session to gain access.

"auth-mask" 0 is the default.

  - b0 is set if you are logged in as a user at all.
  - b1 is set if you are logged in with the user configured to be admin
  - b2 is set if the account has been verified (the account configured for admin is always verified)
  - b3 is set if your session just did the forgot password flow successfully

```
	      {
	        # things in here can always be served
	        "mountpoint": "/lwsgs",
	        "origin": "file:///usr/share/libwebsockets-test-server/generic-sessions",
	        "origin": "callback://protocol-lws-messageboard",
	        "default": "generic-sessions-login-example.html",
	        "auth-mask": "0",
	        "interpret": {
	                ".js": "protocol-lws-messageboard"
	        }
	       }, {
	        # things in here can only be served if logged in as a user
	        "mountpoint": "/lwsgs/needauth",
	        "origin": "file:///usr/share/libwebsockets-test-server/generic-sessions/needauth",
	        "origin": "callback://protocol-lws-messageboard",
	        "default": "generic-sessions-login-example.html",
	        "auth-mask": "5", # logged in as a verified user
	        "interpret": {
	                ".js": "protocol-lws-messageboard"
	        }
	       }, {
	        # things in here can only be served if logged in as admin
	        "mountpoint": "/lwsgs/needadmin",
	        "origin": "file:///usr/share/libwebsockets-test-server/generic-sessions/needadmin",
	        "origin": "callback://protocol-lws-messageboard",
	        "default": "generic-sessions-login-example.html",
	        "auth-mask": "7", # b2 = verified (by email / or admin), b1 = admin, b0 = logged in with any user name
	        "interpret": {
	                ".js": "protocol-lws-messageboard"
	        }
	       }
```
Note that the name of the real application protocol that uses generic-sessions
is used, not generic-sessions itself.

The vhost configures the storage dir, admin credentials and session cookie lifetimes:

```
	     "ws-protocols": [{
	       "protocol-generic-sessions": {
	         "status": "ok",
	         "admin-user": "admin",

	# create the pw hash like this (for the example pw, "jipdocesExunt" )
	# $ echo -n "jipdocesExunt" | sha1sum
	# 046ce9a9cca769e85798133be06ef30c9c0122c9 -
	#
	# Obviously ** change this password hash to a secret one before deploying **
	#
	         "admin-password-sha1": "046ce9a9cca769e85798133be06ef30c9c0122c9",
	         "session-db": "/var/www/sessions/lws.sqlite3",
	         "timeout-idle-secs": "600",
		 "timeout-anon-idle-secs": "1200",
	         "timeout-absolute-secs": "6000",
	# the confounder is part of the salted password hashes.  If this config
	# file is in a 0700 root:root dir, an attacker with apache credentials
	# will have to get the confounder out of the process image to even try
	# to guess the password hashes.
	         "confounder": "Change to <=31 chars of junk",

	         "email-from": "noreply@example.com",
	         "email-smtp-ip": "127.0.0.1",
	         "email-expire": "3600",
	         "email-helo": "myhost.com",
	         "email-contact-person": "Set Me <real-person@email.com>",
	         "email-confirm-url-base": "http://localhost:7681/lwsgs"
	       }
```

The email- related settings control generation of automatic emails for
registration and forgotten password.

 - `email-from`: The email address automatic emails are sent from

 - `email-smtp-ip`: Normally 127.0.0.1, if you have a suitable server on port
   25 on your lan you can use this instead here.

 - `email-expire`: Seconds that links sent in email will work before being
   deleted

 - `email-helo`: HELO to use when communicating with your SMTP server

 - `email-contact-person`: mentioned in the automatic emails as a human who can
   answer questions

 - `email-confirm-url-base`: the URL to start links with in the emails, so the
   recipient can get back to the web server

The real protocol that makes use of generic-sessions must also be listed and
any configuration it needs given

```
	       "protocol-lws-messageboard": {
	         "status": "ok",
	         "message-db": "/var/www/sessions/messageboard.sqlite3"
	       },
```

Notice the real application uses his own sqlite db, no details about how
generic-sessions works or how it stores data are available to it.


@section gspwc Lwsgs Password Confounder

You can also define a per-vhost confounder shown in the example above, used
when aggregating the password with the salt when it is hashed.  Any attacker
will also need to get the confounder along with the database, which you can
make harder by making the config dir only eneterable / readable by root.


@section gsprep Lwsgs Preparing the db directory

You will have to prepare the db directory so it's suitable for the lwsws user to use,
that usually means apache, eg

```
	# mkdir -p /var/www/sessions
	# chown root:apache /var/www/sessions
	# chmod 770 /var/www/sessions
```

@section gsrmail Lwsgs Email configuration

lwsgs will can send emails by talking to an SMTP server on localhost:25.  That
will usually be sendmail or postfix, you should confirm that works first by
itself using the `mail` application to send on it.

lwsgs has been tested on stock Fedora sendmail and postfix.


@section gsap Lwsgs Integration with another protocol

lwsgs is designed to provide sessions and accounts in a standalone and generic way.

But it's not useful by itself, there will always be the actual application who wants
to make use of generic-sessions features.

We provide the "messageboard" plugin as an example of how to integrate with
your actual application protocol.

The basic approach is the 'real' protocol handler (usually a plugin itself)
subclasses the generic-sessions plugin and calls through to it by default.

The "real" protocol handler entirely deals with ws-related stuff itself, since
generic-sessions does not use ws.  But for

 - LWS_CALLBACK_HTTP
 - LWS_CALLBACK_HTTP_BODY
 - LWS_CALLBACK_HTTP_BODY_COMPLETION
 - LWS_CALLBACK_HTTP_DROP_PROTOCOL

the "real" protocol handler checks if it recognizes the activity (eg, his own
POST form URL) and if not, passes stuff through to the generic-sessions protocol callback to handle it.  To simplify matters the real protocol can just pass
through any unhandled messages to generic-sessions.

The "real" protocol can get a pointer to generic-sessions protocol on the
same vhost using

```
	vhd->gsp = lws_vhost_name_to_protocol(vhd->vh, "protocol-generic-sessions");
```

The "real" protocol must also arrange generic-sessions per_session_data in his
own per-session allocation.  To allow keeping generic-sessions opaque, the
real protocol must allocate that space at runtime, using the pss size
the generic-sessions protocol struct exposes

```
	struct per_session_data__myapp {
		void *pss_gs;
	...

		pss->pss_gs = malloc(vhd->gsp->per_session_data_size);
```

The allocation reserved for generic-sessions is then used as user_space when
the real protocol calls through to the generic-sessions callback

```
	vhd->gsp->callback(wsi, reason, &pss->pss_gs, in, len);
```

In that way the "real" protocol can subclass generic-sessions functionality.


To ease management of these secondary allocations, there are callbacks that
occur when a wsi binds to a protocol and when the binding is dropped.  These
should be used to malloc and free and kind of per-connection
secondary allocations.

```
	case LWS_CALLBACK_HTTP_BIND_PROTOCOL:
		if (!pss || pss->pss_gs)
			break;

		pss->pss_gs = malloc(vhd->gsp->per_session_data_size);
		if (!pss->pss_gs)
			return -1;

		memset(pss->pss_gs, 0, vhd->gsp->per_session_data_size);
		break;

	case LWS_CALLBACK_HTTP_DROP_PROTOCOL:
		if (vhd->gsp->callback(wsi, reason, pss ? pss->pss_gs : NULL, in, len))
			return -1;

		if (pss->pss_gs) {
			free(pss->pss_gs);
			pss->pss_gs = NULL;
		}
		break;
```


#section gsapsib Getting session-specific information from another protocol

At least at the time when someone tries to upgrade an http(s) connection to
ws(s) with your real protocol, it is necessary to confirm the cookie the http(s)
connection has with generic-sessions and find out his username and other info.

Generic sessions lets another protocol check it again by calling his callback,
and lws itself provides a generic session info struct to pass the related data

```
	struct lws_session_info {
		char username[32];
		char email[100];
		char ip[72];
		unsigned int mask;
		char session[42];
	};

	struct lws_session_info sinfo;
	...
	vhd->gsp->callback(wsi, LWS_CALLBACK_SESSION_INFO,
				   &pss->pss_gs, &sinfo, 0);
```

After the call to generic-sessions, the results can be

 -  all the strings will be zero-length and .mask zero, there is no usable cookie

  - only .ip and .session are set: the cookie is OK but no user logged in

  - all the strings contain information about the logged-in user

the real protocol can use this to reject attempts to open ws connections from
http connections that are not authenticated; afterwards there's no need to
check the ws connection auth status again.

Notes about generic-table
=========================

@section gtint What is generic-table?

Generic-table is a JSON schema and client-side JS file that makes it easy to
display live, table structured HTML over a ws link.

An example plugin and index.html using it are provided, but lwsgt itself doesn't
have its own plugin, it's just a JSON schema and client-side JS that other
plugins can use to simplify displaying live, table-based data without having
to reinvent the wheel each time.

The ws protocol sends JSON describing the table, and then JSON updating the table
contents when it chooses, the brower table is updated automatically, live.

\image html lwsgt-overview.png

 - Example protocol plugin (displays directory contents): https://github.com/warmcat/libwebsockets/tree/master/plugins/generic-table/protocol_table_dirlisting.c

 - Example HTML: https://github.com/warmcat/libwebsockets/tree/master/plugins/generic-table/assets/index.html

 - lwsgt.js (client-side table rendering / ws link management): https://github.com/warmcat/libwebsockets/tree/master/plugins/generic-table/assets/lwsgt.js


@section gteb Enabling for build

Enable the demo plugin at CMake with -DLWS_WITH_PLUGINS=1


@section gtinth Integrating with your html

 - In your HEAD section, include lwsgt.js

```
	<script src="lwsgt.js"></script>
```

 - Also in your HEAD section, style the lwsgt CSS, eg

```
	<style>
	.lwsgt_title { font-size: 24; text-align:center }
	.lwsgt_breadcrumbs { font-size: 18; text-align:left }
	.lwsgt_table { font-size: 14; padding:12px; margin: 12px; align:center }
	.lwsgt_hdr { font-size: 18; text-align:center;
		     background-color: rgba(40, 40, 40, 0.8); color: white }
	.lwsgt_tr { padding: 10px  }
	.lwsgt_td { padding: 3px  }
	</style>
```

You can skip this but the result will be less beautiful until some CSS is
provided.

 - In your body section, declare a div with an id (can be whatever you want)

```
	<tr><td><div id="lwsgt1" class="group1"></div></td></tr>
```

lwsgt JS will put its content there.

 - Finally in a <script> at the end of your page, instantiate lwsgt and
provide a custom callback for clickable links

```
	<script>
	var v1 = new lwsgt_initial("Dir listing demo",
				   "protocol-lws-table-dirlisting",
				   "lwsgt1", "lwsgt_dir_click", "v1");

	function lwsgt_dir_click(gt, u, col, row)
	{
		if (u[0] == '=') { /* change directory */
			window[gt].lwsgt_ws.send(u.substring(1, u.length));
			return;
		}
		var win = window.open(u, '_blank');
	  	win.focus();
	}

  	</script>
```

In the callback, you can recover the ws object by `window[gt].lwsgt_ws`.


@section gtc Lwsgt constructor

To instantiate the ws link and lwsgt instance, your HTML must call a lwsgt
constructor for each region on the page managed by lwsgt.

`var myvar = new lwsgt_initial(title, ws_protocol, div_id, click_cb, myvar);`

All of the arguments are strings.

| Parameter       | Description                                             |
|-----------------|---------------------------------------------------------|
| title           | Title string to go above the table                      |
| ws_protocol     | Protocol name string to use when making ws connection   |
| div_id          | HTML id of div to fill with content                     |
| click_cb        | Callback function name string to handle clickable links |
| myvar           | Name of var used to hold this instantiation globally    |

Note "myvar" is needed so it can be passed to the click handling callback.


@section gtclick Lwsgt click handling function

When a clickable link produced by lwsgt is clicked, the function named in the
click_cb parameter to lwsgt_initial is called.

That function is expected to take four parameters, eg

`function lwsgt_dir_click(gt, u, col, row)`

| Parameter | Description                                               |
|------- ---|-----------------------------------------------------------|
| gt        | Name of global var holding this lwsgt context (ie, myvar) |
| u         | Link "url" string                                         |
| col       | Table column number link is from                          |
| row       | Table row number link is from                             |



@section gtgj Generic-table JSON

### Column layout

When the ws connection is established, the protocol should send a JSON message
describing the table columns.  For example

```
	  "cols": [
		{ "name": "Date" },
		{ "name": "Size", "align": "right" },
		{ "name": "Icon" },
		{ "name": "Name", "href": "uri"},
		{ "name": "uri", "hide": "1" }
	    ]
	  }
```

 - This describes 5 columns

 - Only four columns (not "uri") should be visible

 - "Name" should be presented as a clickable link using "uri" as the
   destination, when a "uri" field is presented.

 - "Size" field should be presented aligned to the right

 ### Breadcrumbs

 When a view is hierarchical, it's useful to provide a "path" with links back
 in the "path", known as "breadcrumbs".

 Elements before the last one should provide a "url" member as well as the
 displayable name, which is used to create the link destination.

 The last element, being the current displayed page should not have a url
 member and be displayed without link style.


 ```
 	"breadcrumbs":[{"name":"top", "url": "/" }, {"name":"mydir"}]
 ```

 ### Table data

 The actual file data consists of an array of rows, containing the columns
 mentioned in the original "cols" section.

 ```
 	"data":[
 		{
 		 "Icon":" ",
 		 "Date":"2015-Feb-06 03:08:35 +0000",
 		 "Size":"1406",
 		 "uri":"./serve//favicon.ico",
 		 "Name":"favicon.ico"
 		}
 	]

 ```

 @section gtdirl Setting up protocol-lws-table-dirlisting

 The example protocol needs two mounts, one to provide the index.html, js and
 the protocol itself

 ```
 	{
	 "mountpoint": "/dirtest",
         "origin": "file:///usr/share/libwebsockets-test-server/generic-table",
	 "origin": "callback://protocol-lws-table-dirlisting",
	 "default": "index.html",
	 "pmo": [{
		"dir": "/usr/share/libwebsockets-test-server"
	 }]
	},
```

The protocol wants a per-mount option (PMO) to tell it the base directory it
is serving from, named "dir".

The other mount is there to simply serve items that get clicked on from the
table in a secure way

```
	{
	 "mountpoint": "/dirtest/serve",
         "origin": "file:///usr/share/libwebsockets-test-server",
	 "default": "index.html"
	},
```

This last bit is not related to using lwsgt itself.

# h2 long poll in lws

lws server and client can support "immortal" streams that are
not subject to normal timeouts under a special condition.  These
are read-only (to the client).

Network connections that contain at least one immortal stream
are themselves not subject to timeouts until the last immortal
stream they are carrying closes.

Because of this, it's recommended there is some other way of
confirming that the client is still active.

## Setting up lws server for h2 long poll

Vhosts that wish to allow clients to serve these immortal
streams need to set the info.options flag `LWS_SERVER_OPTION_VH_H2_HALF_CLOSED_LONG_POLL`
at vhost creation time.  The JSON config equivalent is to set

```
"h2-half-closed-long-poll": "1"
```

on the vhost.  That's all that is needed.

Streams continue to act normally for timeout with the exception
client streams are allowed to signal they are half-closing by
sending a zero-length DATA frame with END_STREAM set.  These
streams are allowed to exist outside of any timeout and data
can be sent on them at will in the server -> client direction.

## Setting client streams for long poll

An API is provided to allow established h2 client streams to
transition to immortal mode and send the END_STREAM to the server
to indicate it.

```
int
lws_h2_client_stream_long_poll_rxonly(struct lws *wsi);
```

## Example applications

You can confirm the long poll flow simply using example applications.
Build and run `http-server/minimal-http-server-h2-long-poll` in one
terminal.

In another, build the usual `http-client/minimal-http-client` example
and run it with the flags `-l --long-poll`

The client will connect to the server and transition to the immortal mode.
The server sends a timestamp every minute to the client, and that will
stay up without timeouts.

# Http fallback and raw proxying

Lws has several interesting options and features that can be applied to get
some special behaviours... this article discusses them and how they work.

## Overview of normal vhost selection

Lws supports multiple http or https vhosts sharing a listening socket on the
same port.

For unencrypted http, the Host: header is used to select which vhost the
connection should bind to, by comparing what is given there against the
names the server was configured with for the various vhosts.  If no match, it
selects the first configured vhost.

For TLS, it has an extension called SNI (Server Name Indication) which tells
the server early in the TLS handshake the host name the connection is aimed at.
That allows lws to select the vhost early, and use vhost-specific TLS certs
so everything is happy.  Again, if there is no match the connection proceeds
using the first configured vhost and its certs.

## Http(s) fallback options

What happens if you try to connect, eg, an ssh client to the http server port
(this is not an idle question...)?  Obviously the http server part or the tls
part of lws will fail the connection and close it.  (We will look at that flow
in a moment in detail for both unencrypted and tls listeners.)

However if the first configured vhost for the port was created with the
vhost creation info struct `.options` flag `LWS_SERVER_OPTION_FALLBACK_TO_APPLY_LISTEN_ACCEPT_CONFIG`,
then instead of the error, the connection transitions to whatever role was
given in the vhost creation info struct `.listen_accept_role` and `.listen_accept_protocol`.

With lejp-conf / lwsws, the options can be applied to the first vhost using:

```
   "listen-accept-role": "the-role-name",
   "listen-accept-protocol": "the-protocol-name",
   "fallback-listen-accept": "1"
```

See `./minimal-examples/raw/minimal-raw-fallback-http-server` for examples of
all the options in use via commandline flags.

So long as the first packet for the protocol doesn't look like GET, POST, or
a valid tls packet if connection to an https vhost, this allows the one listen
socket to handle both http(s) and a second protocol, as we will see, like ssh.

Notice there is a restriction that no vhost selection processing is possible,
neither for tls listeners nor plain http ones... the packet belonging to a
different protocol will not send any Host: header nor tls SNI.

Therefore although the flags and settings are applied to the first configured
vhost, actually their effect is global for a given listen port.  If enabled,
all vhosts on the same listen port will do the fallback action.

### Plain http flow

![plain http flow](/doc-assets/accept-flow-1.svg)

Normally, if the first received packet does not contain a valid HTTP method,
then the connection is dropped.  Which is what you want from an http server.

However if enabled, the connection can transition to the defined secondary
role / protocol.

|Flag|lejp-conf / lwsws|Function|
|---|---|---|
|`LWS_SERVER_OPTION_FALLBACK_TO_APPLY_LISTEN_ACCEPT_CONFIG`|`"fallback-listen-accept": "1"`|Enable fallback processing|

### TLS https flow

![tls https flow](/doc-assets/accept-flow-2.svg)

If the port is listening with tls, the point that a packet from a different
protocol will fail is earlier, when the tls tunnel is being set up.

|Flag|lejp-conf / lwsws|Function|
|---|---|---|
|`LWS_SERVER_OPTION_FALLBACK_TO_APPLY_LISTEN_ACCEPT_CONFIG`|`"fallback-listen-accept": "1"`|Enable fallback processing|
|`LWS_SERVER_OPTION_REDIRECT_HTTP_TO_HTTPS`|`"redirect-http": "1"`|Treat invalid tls packet as http, issue http redirect to https://|
|`LWS_SERVER_OPTION_ALLOW_HTTP_ON_HTTPS_LISTENER`|`"allow-http-on-https": "1"`|Accept unencrypted http connections on this tls port (dangerous)|

The latter two options are higher priority than, and defeat, the first one.

### Non-http listener

![non-http flow](/doc-assets/accept-flow-3.svg)

It's also possible to skip the fallback processing and just force the first
vhost on the port to use the specified role and protocol in the first place.

|Flag|lejp-conf / lwsws|Function|
|---|---|---|
|LWS_SERVER_OPTION_ADOPT_APPLY_LISTEN_ACCEPT_CONFIG|`"apply-listen-accept": "1"`|Force vhost to use listen-accept-role / listen-accept-protocol|

## Using http(s) fallback with raw-proxy

If enabled for build with `cmake .. -DLWS_ROLE_RAW_PROXY=1 -DLWS_WITH_PLUGINS=1`
then lws includes ready-to-use support for raw tcp proxying.

This can be used standalone on the first vhost on a port, but most intriguingly
it can be specified as the fallback for http(s)...

See `./minimal-examples/raw/minimal-raw-proxy-fallback.c` for a working example.

### fallback with raw-proxy in code

On the first vhost for the port, specify the required "onward" pvo to configure
the raw-proxy protocol...you can adjust the "ipv4:127.0.0.1:22" to whatever you
want...

```
	static struct lws_protocol_vhost_options pvo1 = {
	        NULL,
	        NULL,
	        "onward",		/* pvo name */
	        "ipv4:127.0.0.1:22"	/* pvo value */
	};

	static const struct lws_protocol_vhost_options pvo = {
	        NULL,           	/* "next" pvo linked-list */
	        &pvo1,			/* "child" pvo linked-list */
	        "raw-proxy",		/* protocol name we belong to on this vhost */
	        ""              	/* ignored */
	};
```

... and set up the fallback enable and bindings...

```
	info.options |= LWS_SERVER_OPTION_FALLBACK_TO_APPLY_LISTEN_ACCEPT_CONFIG;
	info.listen_accept_role = "raw_proxy";
	info.listen_accept_proxy = "raw_proxy";
	info.pvo = &pvo;
```

### fallback with raw-proxy in JSON conf

On the first vhost for the port, enable the raw-proxy protocol on the vhost and
set the pvo config

```
                "ws-protocols": [{
                        "raw-proxy": {
                         "status": "ok",
                         "onward": "ipv4:127.0.0.1:22"
                        }
                 }],
```

Enable the fallback behaviour on the vhost and the role / protocol binding

```
	"listen-accept-role": "raw-proxy",
	"listen-accept-protocol": "raw-proxy",
	"fallback-listen-accept": "1"
```

### Testing

With this configured, the listen port will function normally for http or https
depending on how it was set up.

But if you try to connect to it with an ssh client, that will also work fine.

The libwebsockets.org server is set up in this way, you can confirm it by
visiting `https://libwebsockets.org` on port 443 as usual, but also trying
`ssh -p 443 invalid@libwebsockets.org`... you will get permission denied from
your ssh client.  With valid credentials in fact that works perfectly for
ssh, scp, git-over-ssh etc all on port 443...

# lws_dll Doubly-linked list

## Introduction

Lws supports two kinds of doubly-linked list, `lws_dll` and `lws_dll2`.

Unless memory is at a big premium, or it has to work on lws < v3.2, it's
best to simply use `lws_dll2`.

![lws_dll overview](../doc-assets/lws_dll.svg)

## How to use

The basics are the same for lws_dll and lws_dll2.

The list objects point only to themselves, and you use the `lws_container_of`
macro to get a pointer to your struct that contains the list object.  Doing
it this way

 - the list object does not have to be the first thing in your struct

 - your struct can contain multiple list objects and appear on lists belonging
   to multiple owners simultaenously,

### lws_dll Minimal example

```
struct mystruct {
	....
	lws_dll list;
	...
};

lws_dll owner;
```

Adding a mystruct to the owner list (...add_tail() works the same way but adds
to the other end of the list)

```
	struct mystruct *p;

	...

	lws_dll_add_head(&p->list, &owner);
```

Removing the list object from its owner

```
	lws_dll2_remove(&p->list, &owner);
```

If you have a `struct lws_dll *d` pointing to `list` in struct mystruct, you can
convert it to a `struct mystruct *p` ike this

```
	struct mystruct *p = lws_container_of(d, struct lws_dll, list);
```

### lws_dll2 Minimal example


```
struct mystruct {
	....
	lws_dll2 list;
	...
};

lws_dll2_owner owner;
```

Adding a mystruct to the owner list (...add_tail() works the same way but adds
to the other end of the list)

```
	struct mystruct *p;

	...

	lws_dll2_add_head(&p->list, &owner);
```

Removing the list object from its owner (notice compared to lws_dll, it doesn't
need to be told the owner)

```
	lws_dll2_remove(&p->list);
```

If you have a `struct lws_dll2 *d` pointing to `list` in struct mystruct, you
can convert it to a `struct mystruct *p` ike this

```
	struct mystruct *p = lws_container_of(d, struct lws_dll2, list);
```

## Summary Comparing lws_dll and lws_dll2

 - both offer a doubly-linked list object, and (since v3.2) track both the
   head and tail in an "list owner" object

 - both are initalized by memsetting to 0

 - for `lws_dll`, it reuses an `lws_dll` as the "owner", for `lws_dll2`, there's a
   specific `lws_dll2_owner` structure for that

 - `lws_dll2_owner` also keeps count of the number of list elements

 - `lws_dll2` knows which owner's list it is participating on.  So it can remove
   itself and update the owner without the caller needing to know its owner.
   In the case there are several potential owners list objects may be on, this
   is very convenient.

 - `lws_dll` is simpler and has a smaller footprint (two pointers per entry vs
   three).  But you have to know the exact list owner to perform operations on
   it.

## apis

|function|lws_dll|lws_dll2|
|---|---|---|
|add entry at head|`void lws_dll_add_head(struct lws_dll *d, struct lws_dll *phead)`|`void lws_dll2_add_head(struct lws_dll2 *d, struct lws_dll2_owner *owner)`|
|add entry at tail|`void lws_dll_add_tail(struct lws_dll *d, struct lws_dll *phead);`|`void lws_dll2_add_tail(struct lws_dll2 *d, struct lws_dll2_owner *owner)`|
|remove entry from its owning list|`void lws_dll_remove_track_tail(struct lws_dll *d, struct lws_dll *phead)`|`void lws_dll2_add_tail(struct lws_dll2 *d, struct lws_dll2_owner *owner)`|
|get owner|(not supported)|`struct lws_dll2_owner * lws_dll2_owner(const struct lws_dll2 *d)`|
|check if item is detached from any list|`lws_dll_is_detached(struct lws_dll *d, struct lws_dll *phead)|int lws_dll2_is_detached(const struct lws_dll2 *d)`|
|iterate through items on list|`int lws_dll_foreach_safe(struct lws_dll *phead, void *user, int (*cb)(struct lws_dll *d, void *user))|int lws_dll2_foreach_safe(struct lws_dll2_owner *owner, void *user, int (*cb)(struct lws_dll2 *d, void *user))`|

# `lws_retry_bo_t` client connection management

This struct sets the policy for delays between retries, and for
how long a connection may be 'idle' before it first tries to
ping / pong on it to confirm it's up, or drops the connection
if still idle.

## Retry rate limiting

You can define a table of ms-resolution delays indexed by which
connection attempt number is ongoing, this is pointed to by
`.retry_ms_table` with `.retry_ms_table_count` containing the
count of table entries.

`.conceal_count` is the number of retries that should be allowed
before informing the parent that the connection has failed.  If it's
greater than the number of entries in the table, the last entry is
reused for the additional attempts.

`.jitter_percent` controls how much additional random delay is
added to the actual interval to be used... this stops a lot of
devices all synchronizing when they try to connect after a single
trigger event and DDoS-ing the server.

The struct and apis are provided for user implementations, lws does
not offer reconnection itself.

## Connection validity management

Lws has a sophisticated idea of connection validity and the need to
reconfirm that a connection is still operable if proof of validity
has not been seen for some time.  It concerns itself only with network
connections rather than streams, for example, it only checks h2
network connections rather than the individual streams inside (which
is consistent with h2 PING frames only working at the network stream
level itself).

Connections may fail in a variety of ways, these include that no traffic
at all is passing, or, eg, incoming traffic may be received but no
outbound traffic is making it to the network, and vice versa.  In the
case that tx is not failing at any point but just isn't getting sent,
endpoints can potentially kid themselves that since "they are sending"
and they are seeing RX, the combination means the connection is valid.
This can potentially continue for a long time if the peer is not
performing keepalives.

"Connection validity" is proven when one side sends something and later
receives a response that can only have been generated by the peer
receiving what was just sent.  This can happen for some kinds of user
transactions on any stream using the connection, or by sending PING /
PONG protocol packets where the PONG is only returned for a received PING.

To ensure that the generated traffic is only sent when necessary, user
code can report for any stream that it has observed a transaction amounting
to a proof of connection validity using an api.  This resets the timer for
the associated network connection before the validity is considered
expired.

`.secs_since_valid_ping` in the retry struct sets the number of seconds since
the last validity after which lws will issue a protocol-specific PING of some
kind on the connection.  `.secs_since_valid_hangup` specifies how long lws
will allow the connection to go without a confirmation of validity before
simply hanging up on it.

## Defaults

The context defaults to having a 5m valid ping interval and 5m10s hangup interval,
ie, it'll send a ping at 5m idle if the protocol supports it, and if no response
validating the connection arrives in another 10s, hang up the connection.

User code can set this in the context creation info and can individually set the
retry policy per vhost for server connections.  Client connections can set it
per connection in the client creation info `.retry_and_idle_policy`.

## Checking for h2 and ws

Check using paired minimal examples with the -v flag on one or both sides to get a
small validity check period set of 3s / 10s

Also give, eg, -d1039 to see info level debug logging

### h2

```
$ lws-minimal-http-server-h2-long-poll -v

$ lws-minimal-http-client -l -v
```

### ws

```
$ lws-minimal-ws-server-h2 -s -v

$ lws-minimal-ws-client-ping -n --server 127.0.0.1 --port 7681 -v
```

# `struct lws_sequencer` introduction

Often a single network action like a client GET is just part of a
larger series of actions, perhaps involving different connections.

Since lws operates inside an event loop, if the outer sequencing
doesn't, it can be awkward to synchronize these steps with what's
happening on the network with a particular connection on the event
loop thread.

![lws_sequencer](/doc-assets/lws_sequencer.svg)

`struct lws_sequencer` provides a generic way to stage multi-step
operations from inside the event loop.  Because it participates
in the event loop similar to a wsi, it always operates from the
service thread context and can access structures that share the
service thread without locking.  It can also provide its own
higher-level timeout handling.

Naturally you can have many of them running in the same event
loop operating independently.

Sequencers themselves bind to a pt (per-thread) service thread,
by default there's only one of these and it's the same as saying
they bind to an `lws_context`.  The sequencer callback may create
wsi which in turn are bound to a vhost, but the sequencer itself
is above all that.

## Sequencer timeouts

The sequencer additionally maintains its own second-resolution timeout
checked by lws for the step being sequenced... this is independent of
any lws wsi timeouts which tend to be set and reset for very short-term
timeout protection inside one transaction.

The sequencer timeout operates separately and above any wsi timeout, and
is typically only reset by the sequencer callback when it receives an
event indicating a step completed or failed, or it sets up the next sequence
step.

If the sequencer timeout expires, then the sequencer receives a queued
`LWSSEQ_TIMED_OUT` message informing it, and it can take corrective action
or schedule a retry of the step.  This message is queued and sent normally
under the service thread context and in order of receipt.

Unlike lws timeouts which force the wsi to close, the sequencer timeout
only sends the message.  This allows the timeout to be used to, eg, wait
out a retry cooloff period and then start the retry when the
`LWSSEQ_TIMED_OUT` is received, according to the state of the sequencer.

## Creating an `struct lws_sequencer`

```
typedef struct lws_seq_info {
	struct lws_context		*context;   /* lws_context for seq */
	int				tsi;	    /* thread service idx */
	size_t				user_size;  /* size of user alloc */
	void				**puser;    /* place ptr to user */
	lws_seq_event_cb		cb;	    /* seq callback */
	const char			*name;	    /* seq name */
	const lws_retry_bo_t		*retry;	    /* retry policy */
} lws_seq_info_t;
```

```
struct lws_sequencer *
lws_sequencer_create(lws_seq_info_t *info);
```

When created, in lws the sequencer objects are bound to a 'per-thread',
which is by default the same as to say bound to the `lws_context`.  You
can tag them with an opaque user data pointer, and they are also bound to
a user-specified callback which handles sequencer events

```
typedef int (*lws_seq_event_cb)(struct lws_sequencer *seq, void *user_data,
				lws_seq_events_t event, void *data);
```

`struct lws_sequencer` objects are private to lws and opaque to the user.  A small
set of apis lets you perform operations on the pointer returned by the
create api.

## Queueing events on a sequencer

Each sequencer object can be passed "events", which are held on a per-sequencer
queue and handled strictly in the order they arrived on subsequent event loops.
`LWSSEQ_CREATED` and `LWSSEQ_DESTROYED` events are produced by lws reflecting
the sequencer's lifecycle, but otherwise the event indexes have a user-defined
meaning and are queued on the sequencer by user code for eventual consumption
by user code in the sequencer callback.

Pending events are removed from the sequencer queues and sent to the sequencer
callback from inside the event loop at a rate of one per event loop wait.

## Destroying sequencers

`struct lws_sequencer` objects are cleaned up during context destruction if they are
still around.

Normally the sequencer callback receives a queued message that
informs it that it's either failed at the current step, or succeeded and that
was the last step, and requests that it should be destroyed by returning
`LWSSEQ_RET_DESTROY` from the sequencer callback.

## Lifecycle considerations

Sequencers may spawn additional assets like client wsi as part of the sequenced
actions... the lifecycle of the sequencer and the assets overlap but do not
necessarily depend on each other... that is a wsi created by the sequencer may
outlive the sequencer.

It's important therefore to detach assets from the sequencer and the sequencer
from the assets when each step is over and the asset is "out of scope" for the
sequencer.  It doesn't necessarily mean closing the assets, just making sure
pointers are invalidated.  For example, if a client wsi held a pointer to the
sequencer as its `.user_data`, when the wsi is out of scope for the sequencer
it can set it to NULL, eg, `lws_set_wsi_user(wsi, NULL);`.

Under some conditions wsi may want to hang around a bit to see if there is a
subsequent client wsi transaction they can be reused on.  They will clean
themselves up when they time out.

## Watching wsi lifecycle from a sequencer

When a sequencer is creating a wsi as part of its sequence, it will be very
interested in lifecycle events.  At client wsi creation time, the sequencer
callback can set info->seq to itself in order to receive lifecycle messages
about its wsi.

|message|meaning|
|---|---|
|`LWSSEQ_WSI_CONNECTED`|The wsi has become connected|
|`LWSSEQ_WSI_CONN_FAIL`|The wsi has failed to connect|
|`LWSSEQ_WSI_CONN_CLOSE`|The wsi had been connected, but has now closed|

By receiving these, the sequencer can understand when it should attempt
reconnections or that it cannot progress the sequence.

When dealing with wsi that were created by the sequencer, they may close at
any time, eg, be closed by the remote peer or an intermediary.  The
`LWSSEQ_WSI_CONN_CLOSE` message may have been queued but since they are
strictly handled in the order they arrived, before it was
handled an earlier message may want to cause some api to be called on
the now-free()-d wsi.  To detect this situation safely, there is a
sequencer api `lws_sequencer_check_wsi()` which peeks the message
buffer and returns nonzero if it later contains an `LWSSEQ_WSI_CONN_CLOSE`
already.

# lws_struct

## Overview

lws_struct provides a lightweight method for serializing and deserializing C
structs to and from JSON, and to and from sqlite3.

![lws_struct overview](../doc-assets/lws_struct-overview.svg)

 - you provide a metadata array describing struct members one-time, then call
   generic apis to serialize and deserialize

 - supports flat structs, single child struct pointers, and unbounded arrays /
   linked-lists of child objects automatically using [lws_dll2 linked-lists](./README.lws_dll.md)

 - supports boolean and C types char, int, long, long long in explicitly signed
   and unsigned forms

 - supports both char * type string members where the unbounded content is
   separate and pointed to, and fixed length char array[] type members where
   the content is part of the struct

 - huge linear strings are supported by storing to a temp lwsac of chained chunks,
   which is written into a single linear chunk in the main lwsac once the
   total string length is known

 - deserialization allocates into an [lwsac](../lib/misc/lwsac/README.md), so everything is inside as few
   heap allocations as possible while still able to expand to handle arbitrary
   array or strins sizes

 - when deserialized structs are finished with, a single call to free the
   lwsac frees the whole thing without having to walk it

 - stateful serializaton and deserialization allows as-you-get packets incremental
   parsing and production of chunks of as-you-can-send incremental serialization
   output cleanly

## Examples

# `lws_sul` scheduler api

Since v3.2 lws no longer requires periodic checking for timeouts and
other events.  A new system was refactored in where future events are
scheduled on to a single, unified, sorted linked-list in time order,
with everything at us resolution.

This makes it very cheap to know when the next scheduled event is
coming and restrict the poll wait to match, or for event libraries
set a timer to wake at the earliest event when returning to the
event loop.

Everything that was checked periodically was converted to use `lws_sul`
and schedule its own later event.  The end result is when lws is idle,
it will stay asleep in the poll wait until a network event or the next
scheduled `lws_sul` event happens, which is optimal for power.

# Side effect for older code

If your older code uses `lws_service_fd()`, it used to be necessary
to call this with a NULL pollfd periodically to indicate you wanted
to let the background checks happen.  `lws_sul` eliminates the whole
concept of periodic checking and NULL is no longer a valid pollfd
value for this and related apis.

# Using `lws_sul` in user code

See `minimal-http-client-multi` for an example of using the `lws_sul`
scheduler from your own code; it uses it to spread out connection
attempts so they are staggered in time.  You must create an
`lws_sorted_usec_list_t` object somewhere, eg, in you own existing object.

```
static lws_sorted_usec_list_t sul_stagger;
```

Create your own callback for the event... the argument points to the sul object
used when the callback was scheduled.  You can use pointer arithmetic to translate
that to your own struct when the `lws_sorted_usec_list_t` was a member of the
same struct.

```
static void
stagger_cb(lws_sorted_usec_list_t *sul)
{
...
}
```

When you want to schedule the callback, use `lws_sul_schedule()`... this will call
it 10ms in the future

```
	lws_sul_schedule(context, 0, &sul_stagger, stagger_cb, 10 * LWS_US_PER_MS);
```

In the case you destroy your object and need to cancel the scheduled callback, use

```
	lws_sul_schedule(context, 0, &sul_stagger, NULL, LWS_SET_TIMER_USEC_CANCEL);
```

# `lws_system`

See `include/libwebsockets/lws-system.h` for function and object prototypes.

## System integration api

`lws_system` allows you to set a `system_ops` struct at context creation time,
which can write up some function callbacks for system integration.  The goal
is the user code calls these by getting the ops struct pointer from the
context using `lws_system_get_ops(context)` and so does not spread system
dependencies around the user code, making it directly usable on completely
different platforms.

```
typedef struct lws_system_ops {
	int (*reboot)(void);
	int (*set_clock)(lws_usec_t us);
	int (*attach)(struct lws_context *context, int tsi, lws_attach_cb_t cb,
		      lws_system_states_t state, void *opaque,
		      struct lws_attach_item **get);
} lws_system_ops_t;
```

|Item|Meaning|
|---|---|
|`(*reboot)()`|Reboot the system|
|`(*set_clock)()`|Set the system clock|
|`(*attach)()`|Request an event loop callback from another thread context|

### `reboot`

Reboots the device

### `set_clock`

Set the system clock to us-resolution Unix time in seconds

### `attach`

Request a callback from the event loop from a foreign thread.  This is used, for
example, for foreign threads to set up their event loop activity in their
callback, and eg, exit once it is done, with their event loop activity able to
continue wholly from the lws event loop thread and stack context.

## Foreign thread `attach` architecture

When lws is started, it should define an `lws_system_ops_t` at context creation
time which defines its `.attach` handler.  In the `.attach` handler
implementation, it should perform platform-specific locking around a call to
`__lws_system_attach()`, a public lws api that actually queues the callback
request and does the main work.  The platform-specific wrapper is just there to
do the locking so multiple calls from different threads to the `.attach()`
operation can't conflict.

User code can indicate it wants a callback from the lws event loop like this:

```
lws_system_get_ops(context)->attach(context, tsi, cb, state, opaque, NULL)
```

`context` is a pointer to the lws_context, `tsi` is normally 0, `cb` is the user
callback in the form

```
void (*lws_attach_cb_t)(struct lws_context *context, int tsi, void *opaque);
```

`state` is the `lws_system` state we should have reached before performing the
callback (usually, `LWS_SYSTATE_OPERATIONAL`), and `opaque` is a user pointer that
will be passed into the callback.

`cb` will normally want to create scheduled events and set up lws network-related
activity from the event loop thread and stack context.

Once the event loop callback has been booked by calling this api, the thread and
its stack context that booked it may be freed.  It will be called back and can
continue operations from the lws event loop thread and stack context.  For that
reason, if `opaque` is needed it will usually point to something on the heap,
since the stack context active at the time the callback was booked may be long
dead by the time of the callback.

See ./lib/system/README.md for more details.

## `lws_system` blobs

"Blobs" are arbitrary binary objects that have a total length.  Lws lets you set
them in two ways

 - "directly", by pointing to them, which has no heap implication

 - "heap", by adding one or more arbitrary chunk to a chained heap object

In the "heap" case, it can be incrementally defined and the blob doesn't all
have to be declared at once.

For read, the same api allows you to read all or part of the blob into a user
buffer.

The following kinds of blob are defined

|Item|Meaning|
|---|---|
|`LWS_SYSBLOB_TYPE_AUTH`|Auth-related blob 1, typically a registration token|
|`LWS_SYSBLOB_TYPE_AUTH + 1`|Auth-related blob 2, typically an auth token|
|`LWS_SYSBLOB_TYPE_CLIENT_CERT_DER`|Client cert public part|
|`LWS_SYSBLOB_TYPE_CLIENT_KEY_DER`|Client cert key part|
|`LWS_SYSBLOB_TYPE_DEVICE_SERIAL`|Arbitrary device serial number|
|`LWS_SYSBLOB_TYPE_DEVICE_FW_VERSION`|Arbitrary firmware version|
|`LWS_SYSBLOB_TYPE_DEVICE_TYPE`|Arbitrary Device Type identifier|
|`LWS_SYSBLOB_TYPE_NTP_SERVER`|String with the ntp server address (defaults to pool.ntp.org)|

### Blob handle api

Returns an object representing the blob for a particular type (listed above)

```
lws_system_blob_t *
lws_system_get_blob(struct lws_context *context, lws_system_blob_item_t type,
                    int idx);
```

### Blob Setting apis

Sets the blob to point length `len` at `ptr`.  No heap allocation is used.

```
void
lws_system_blob_direct_set(lws_system_blob_t *b, const uint8_t *ptr, size_t len);
```

Allocates and copied `len` bytes from `buf` into heap and chains it on the end of
any existing.

```
int
lws_system_blob_heap_append(lws_system_blob_t *b, const uint8_t *buf, size_t len)
```

Remove any content from the blob, freeing it if it was on the heap

```
void
lws_system_blob_heap_empty(lws_system_blob_t *b)
```

### Blob getting apis

Get the total size of the blob (ie, if on the heap, the aggreate size of all the
chunks that were appeneded)

```
size_t
lws_system_blob_get_size(lws_system_blob_t *b)
```

Copy part or all of the blob starting at offset ofs into a user buffer at buf.
`*len` should be the length of the user buffer on entry, on exit it's set to
the used extent of `buf`.  This works the same whether the bob is a direct pointer
or on the heap.

```
int
lws_system_blob_get(lws_system_blob_t *b, uint8_t *buf, size_t *len, size_t ofs)
```

If you know that the blob was handled as a single direct pointer, or a single
allocation, you can get a pointer to it without copying using this.

```
int
lws_system_blob_get_single_ptr(lws_system_blob_t *b, const uint8_t **ptr)
```

### Blob destroy api

Deallocates any heap allocation for the blob

```
void
lws_system_blob_destroy(lws_system_blob_t *b)
```


## System state and notifiers

Lws implements a state in the context that reflects the readiness of the system
for various steps leading up to normal operation.  By default it acts in a
backwards-compatible way and directly reaches the OPERATIONAL state just after
the context is created.

However other pieces of lws, and user, code may define notification handlers
that get called back when the state changes incrementally, and may veto or delay
the changes until work necessary for the new state has completed asynchronously.

The generic states defined are:

|State|Meaning|
|---|---|
|`LWS_SYSTATE_CONTEXT_CREATED`|The context was just created.|
|`LWS_SYSTATE_INITIALIZED`|The vhost protocols have been initialized|
|`LWS_SYSTATE_IFACE_COLDPLUG`|Existing network interfaces have been iterated|
|`LWS_SYSTATE_DHCP`|Network identity is available|
|`LWS_SYSTATE_TIME_VALID`|The system knows the time|
|`LWS_SYSTATE_POLICY_VALID`|If the system needs information about how to act from the net, it has it|
|`LWS_SYSTATE_REGISTERED`|The device has a registered identity|
|`LWS_SYSTATE_AUTH1`|The device identity has produced a time-limited access token|
|`LWS_SYSTATE_AUTH2`|Optional second access token for different services|
|`LWS_SYSTATE_OPERATIONAL`|The system is ready for user code to work normally|
|`LWS_SYSTATE_POLICY_INVALID`|All connections are being dropped because policy information is changing.  It will transition back to `LWS_SYSTATE_INITIALIZED` and onward to `OPERATIONAL` again afterwards with the new policy|

### Inserting a notifier

You should create an object `lws_system_notify_link_t` in non-const memory and zero it down.
Set the `notify_cb` member and the `name` member and then register it using either
`lws_system_reg_notifier()` or the `.register_notifier_list`
member of the context creation info struct to make sure it will exist early
enough to see all events.  The context creation info method takes a list of
pointers to notify_link structs ending with a NULL entry.

Notes about lwsws
=================

@section lwsws Libwebsockets Web Server

lwsws is an implementation of a very lightweight, ws-capable generic web
server, which uses libwebsockets to implement everything underneath.

If you are basically implementing a standalone server with lws, you can avoid
reinventing the wheel and use a debugged server including lws.


@section lwswsb Build

Just enable -DLWS_WITH_LWSWS=1 at cmake-time.

It enables libuv and plugin support automatically.

NOTICE on Ubuntu, the default libuv package is called "libuv-0.10".  This is ancient.

You should replace this with libuv1 and libuv1-dev before proceeding.

@section lwswsc Lwsws Configuration

lwsws uses JSON config files, they're pure JSON except:

 - '#' may be used to turn the rest of the line into a comment.

 - There's also a single substitution, if a string contains "_lws_ddir_", then that is
replaced with the LWS install data directory path, eg, "/usr/share" or whatever was
set when LWS was built + installed.  That lets you refer to installed paths without
having to change the config if your install path was different.

There is a single file intended for global settings

/etc/lwsws/conf
```
	# these are the server global settings
	# stuff related to vhosts should go in one
	# file per vhost in ../conf.d/

	{
	  "global": {
	   "username": "apache",
	   "groupname": "apache",
	   "count-threads": "1",
	   "server-string": "myserver v1", # returned in http headers
	   "ws-pingpong-secs": "200", # confirm idle established ws connections this often
	   "init-ssl": "yes"
	 }
	}
```
and a config directory intended to take one file per vhost

/etc/lwsws/conf.d/warmcat.com
```
	{
		"vhosts": [{
			"name": "warmcat.com",
			"port": "443",
			"interface": "eth0",  # optional
			"host-ssl-key": "/etc/pki/tls/private/warmcat.com.key",  # if given enable ssl
			"host-ssl-cert": "/etc/pki/tls/certs/warmcat.com.crt",
			"host-ssl-ca": "/etc/pki/tls/certs/warmcat.com.cer",
			"mounts": [{  # autoserve
				"mountpoint": "/",
				"origin": "file:///var/www/warmcat.com",
				"default": "index.html"
			}]
		}]
	}
```
To get started quickly, an example config reproducing the old test server
on port 7681, non-SSL is provided.  To set it up
```
	# mkdir -p /etc/lwsws/conf.d /var/log/lwsws
	# cp ./lwsws/etc-lwsws-conf-EXAMPLE /etc/lwsws/conf
	# cp ./lwsws/etc-lwsws-conf.d-localhost-EXAMPLE /etc/lwsws/conf.d/test-server
	# sudo lwsws
```

@section lwswsacme Using Letsencrypt or other ACME providers

Lws supports automatic provisioning and renewal of TLS certificates.

See ./READMEs/README.plugin-acme.md for examples of how to set it up on an lwsws vhost.

@section lwsogo Other Global Options

 - `reject-service-keywords` allows you to return an HTTP error code and message of your choice
if a keyword is found in the user agent

```
   "reject-service-keywords": [{
        "scumbot": "404 Not Found"
   }]
```

 - `timeout-secs` lets you set the global timeout for various network-related
 operations in lws, in seconds.  It defaults to 5.

@section lwswsv Lwsws Vhosts

One server can run many vhosts, where SSL is in use SNI is used to match
the connection to a vhost and its vhost-specific SSL keys during SSL
negotiation.

Listing multiple vhosts looks something like this
```
	{
	 "vhosts": [ {
	     "name": "localhost",
	     "port": "443",
	     "host-ssl-key":  "/etc/pki/tls/private/libwebsockets.org.key",
	     "host-ssl-cert": "/etc/pki/tls/certs/libwebsockets.org.crt",
	     "host-ssl-ca":   "/etc/pki/tls/certs/libwebsockets.org.cer",
	     "mounts": [{
	       "mountpoint": "/",
	       "origin": "file:///var/www/libwebsockets.org",
	       "default": "index.html"
	       }, {
	        "mountpoint": "/testserver",
	        "origin": "file:///usr/local/share/libwebsockets-test-server",
	        "default": "test.html"
	       }],
	     # which protocols are enabled for this vhost, and optional
	     # vhost-specific config options for the protocol
	     #
	     "ws-protocols": [{
	       "warmcat,timezoom": {
	         "status": "ok"
	       }
	     }]
	    },
	    {
	    "name": "localhost",
	    "port": "7681",
	     "host-ssl-key":  "/etc/pki/tls/private/libwebsockets.org.key",
	     "host-ssl-cert": "/etc/pki/tls/certs/libwebsockets.org.crt",
	     "host-ssl-ca":   "/etc/pki/tls/certs/libwebsockets.org.cer",
	     "mounts": [{
	       "mountpoint": "/",
	       "origin": ">https://localhost"
	     }]
	   },
	    {
	    "name": "localhost",
	    "port": "80",
	     "mounts": [{
	       "mountpoint": "/",
	       "origin": ">https://localhost"
	     }]
	   }

	  ]
	}
```

That sets up three vhosts all called "localhost" on ports 443 and 7681 with SSL, and port 80 without SSL but with a forced redirect to https://localhost


@section lwswsvn Lwsws Vhost name and port sharing

The vhost name field is used to match on incoming SNI or Host: header, so it
must always be the host name used to reach the vhost externally.

 - Vhosts may have the same name and different ports, these will each create a
listening socket on the appropriate port.

 - Vhosts may also have the same port and different name: these will be treated as
true vhosts on one listening socket and the active vhost decided at SSL
negotiation time (via SNI) or if no SSL, then after the Host: header from
the client has been parsed.


@section lwswspr Lwsws Protocols

Vhosts by default have available the union of any initial protocols from context creation time, and
any protocols exposed by plugins.

Vhosts can select which plugins they want to offer and give them per-vhost settings using this syntax
```
	     "ws-protocols": [{
	       "warmcat-timezoom": {
	         "status": "ok"
	       }
	     }]
```

The "x":"y" parameters like "status":"ok" are made available to the protocol during its per-vhost
LWS_CALLBACK_PROTOCOL_INIT (in is a pointer to a linked list of struct lws_protocol_vhost_options
containing the name and value pointers).

To indicate that a protocol should be used when no Protocol: header is sent
by the client, you can use "default": "1"
```
	     "ws-protocols": [{
	       "warmcat-timezoom": {
	         "status": "ok",
	         "default": "1"
	       }
	     }]
```

Similarly, if your vhost is serving a raw protocol, you can mark the protocol
to be selected using "raw": "1"
```
	     "ws-protocols": [{
	       "warmcat-timezoom": {
	         "status": "ok",
	         "raw": "1"
	       }
	     }]
```

See also "apply-listen-accept" below.

@section lwswsovo Lwsws Other vhost options

 - If the three options `host-ssl-cert`, `host-ssl-ca` and `host-ssl-key` are given, then the vhost supports SSL.

 Each vhost may have its own certs, SNI is used during the initial connection negotiation to figure out which certs to use by the server name it's asking for from the request DNS name.

 - `keeplive-timeout` (in secs) defaults to 60 for lwsws, it may be set as a vhost option

 - `interface` lets you specify which network interface to listen on, if not given listens on all.  If the network interface is not usable (eg, ethernet cable out) it will be logged at startup with such vhost not listening, and lws will poll for it and bind a listen socket to the interface if and when it becomes available.

 - "`unix-socket`": "1" causes the unix socket specified in the interface option to be used instead of an INET socket

 - "`unix-socket-perms`": "user:group" allows you to control the unix permissons on the listening unix socket.  It's always get to `0600` mode, but you can control the user and group for the socket fd at creation time.  This allows you to use unix user and groups to control who may open the other end of the unix socket on the local system.

 - "`sts`": "1" causes lwsws to send a Strict Transport Security header with responses that informs the client he should never accept to connect to this address using http.  This is needed to get the A+ security rating from SSL Labs for your server.

 - "`access-log`": "filepath"   sets where apache-compatible access logs will be written

 - `"enable-client-ssl"`: `"1"` enables the vhost's client SSL context, you will need this if you plan to create client conections on the vhost that will use SSL.  You don't need it if you only want http / ws client connections.

 - "`ciphers`": "<cipher list>"  OPENSSL only: sets the allowed list of TLS <= 1.2 ciphers and key exchange protocols for the serving SSL_CTX on the vhost.  The default list is restricted to only those providing PFS (Perfect Forward Secrecy) on the author's Fedora system.

 If you need to allow weaker ciphers, you can provide an alternative list here per-vhost.

 - "`client-ssl-ciphers`": "<cipher list>"  OPENSSL only: sets the allowed list of <= TLS1.2 ciphers and key exchange protocols for the client SSL_CTX on the vhost

 - "`tls13-ciphers`": "<cipher list>"  OPENSSL 1.1.1+ only: sets allowed list of TLS1.3+ ciphers and key exchange protocols for the client SSL_CTX on the vhost.  The default is to allow all.

 - "`client-tls13-ciphers`": "<cipher list>"  OPENSSL 1.1.1+ only: sets the allowed list of TLS1.3+ ciphers and key exchange protocols for the client SSL_CTX on the vhost.  The default is to allow all.

 - "`ecdh-curve`": "<curve name>"   The default ecdh curve is "prime256v1", but you can override it here, per-vhost

 - "`noipv6`": "on"  Disable ipv6 completely for this vhost

 - "`ipv6only`": "on"  Only allow ipv6 on this vhost / "off" only allow ipv4 on this vhost

 - "`ssl-option-set`": "<decimal>"  Sets the SSL option flag value for the vhost.
 It may be used multiple times and OR's the flags together.

 The values are derived from /usr/include/openssl/ssl.h
```
	 # define SSL_OP_NO_TLSv1_1                               0x10000000L
```

 would equate to

```
	 "`ssl-option-set`": "268435456"
 ```
 - "`ssl-option-clear'": "<decimal>"   Clears the SSL option flag value for the vhost.
 It may be used multiple times and OR's the flags together.

 - "`ssl-client-option-set`" and "`ssl-client-option-clear`" work the same way for the vhost Client SSL context

 - "`headers':: [{ "header1": "h1value", "header2": "h2value" }]

allows you to set arbitrary headers on every file served by the vhost

recommended vhost headers for good client security are

```
                   "headers": [{
                        "Content-Security-Policy": "script-src 'self'",
                        "X-Content-Type-Options": "nosniff",
                        "X-XSS-Protection": "1; mode=block",
                        "X-Frame-Options": "SAMEORIGIN"
                 }]

```

 - "`apply-listen-accept`": "on"  This vhost only serves a non-http protocol, specified in "listen-accept-role" and "listen-accept-protocol"

@section lwswsm Lwsws Mounts

Where mounts are given in the vhost definition, then directory contents may
be auto-served if it matches the mountpoint.

Mount protocols are used to control what kind of translation happens

 - file://  serve the uri using the remainder of the url past the mountpoint based on the origin directory.

 Eg, with this mountpoint
```
	       {
	        "mountpoint": "/",
	        "origin": "file:///var/www/mysite.com",
	        "default": "/"
	       }
```
 The uri /file.jpg would serve /var/www/mysite.com/file.jpg, since / matched.

 - ^http:// or ^https://  these cause any url matching the mountpoint to issue a redirect to the origin url

 - cgi://   this causes any matching url to be given to the named cgi, eg
```
	       {
	        "mountpoint": "/git",
	        "origin": "cgi:///var/www/cgi-bin/cgit",
	        "default": "/"
	       }, {
	        "mountpoint": "/cgit-data",
	        "origin": "file:///usr/share/cgit",
	        "default": "/"
	       },
```
 would cause the url /git/myrepo to pass "myrepo" to the cgi /var/www/cgi-bin/cgit and send the results to the client.

 - http:// or https://  these perform reverse proxying, serving the remote origin content from the mountpoint.  Eg

```
		{
		 "mountpoint": "/proxytest",
		 "origin": "https://libwebsockets.org"
		}
```

This will cause your local url `/proxytest` to serve content fetched from libwebsockets.org over ssl; whether it's served from your server using ssl is unrelated and depends how you configured your local server.  Notice if you will use the proxying feature, `LWS_WITH_HTTP_PROXY` is required to be enabled at cmake, and for `https` proxy origins, your lwsws configuration must include `"init-ssl": "1"` and the vhost with the proxy mount must have `"enable-client-ssl": "1"`, even if you are not using ssl to serve.

`/proxytest/abc`, or `/proxytest/abc?def=ghi` etc map to the origin + the part past `/proxytest`, so links and img src urls etc work as do all urls under the origin path.

In addition link and src urls in the document are rewritten so / or the origin url part are rewritten to the mountpoint part.


@section lwswsomo Lwsws Other mount options

1) Some protocols may want "per-mount options" in name:value format.  You can
provide them using "pmo"

	       {
	        "mountpoint": "/stuff",
	        "origin": "callback://myprotocol",
	        "pmo": [{
	                "myname": "myvalue"
	        }]
	       }

2) When using a cgi:// protocol origin at a mountpoint, you may also give cgi environment variables specific to the mountpoint like this
```
	       {
	        "mountpoint": "/git",
	        "origin": "cgi:///var/www/cgi-bin/cgit",
	        "default": "/",
	        "cgi-env": [{
	                "CGIT_CONFIG": "/etc/cgitrc/libwebsockets.org"
	        }]
	       }
```
 This allows you to customize one cgi depending on the mountpoint (and / or vhost).

3) It's also possible to set the cgi timeout (in secs) per cgi:// mount, like this
```
	"cgi-timeout": "30"
```
4) `callback://` protocol may be used when defining a mount to associate a
named protocol callback with the URL namespace area.  For example
```
	       {
	        "mountpoint": "/formtest",
	        "origin": "callback://protocol-post-demo"
	       }
```
All handling of client access to /formtest[anything] will be passed to the
callback registered to the protocol "protocol-post-demo".

This is useful for handling POST http body content or general non-cgi http
payload generation inside a plugin.

See the related notes in README.coding.md

5) Cache policy of the files in the mount can also be set.  If no
options are given, the content is marked uncacheable.
```
	       {
	        "mountpoint": "/",
	        "origin": "file:///var/www/mysite.com",
	        "cache-max-age": "60",      # seconds
	        "cache-reuse": "1",         # allow reuse at client at all
	        "cache-revalidate": "1",    # check it with server each time
	        "cache-intermediaries": "1" # allow intermediary caches to hold
	       }
```

6) You can also define a list of additional mimetypes per-mount
```
	        "extra-mimetypes": {
	                 ".zip": "application/zip",
	                 ".doc": "text/evil"
	         }
```

Normally a file suffix MUST match one of the canned mimetypes or one of the extra
mimetypes, or the file is not served.  This adds a little bit of security because
even if there is a bug somewhere and the mount dirs are circumvented, lws will not
serve, eg, /etc/passwd.

If you provide an extra mimetype entry

			"*": ""

Then any file is served, if the mimetype was not known then it is served without a
Content-Type: header.

7) A mount can be protected by HTTP Basic Auth.  This only makes sense when using
https, since otherwise the password can be sniffed.

You can add a `basic-auth` entry on an http mount like this

```
{
        "mountpoint": "/basic-auth",
        "origin": "file://_lws_ddir_/libwebsockets-test-server/private",
        "basic-auth": "/var/www/balogins-private"
}
```

Before serving anything, lws will signal to the browser that a username / password
combination is required, and it will pop up a dialog.  When the user has filled it
in, lwsws checks the user:password string against the text file named in the `basic-auth`
entry.

The file should contain user:pass one per line

```
testuser:testpass
myuser:hispass
```

The file should be readable by lwsws, and for a little bit of extra security not
have a file suffix, so lws would reject to serve it even if it could find it on
a mount.

After successful authentication, `WSI_TOKEN_HTTP_AUTHORIZATION` contains the
authenticated username.

In the case you want to also protect being able to connect to a ws protocol on
a particular vhost by requiring the http part can authenticate using Basic
Auth before the ws upgrade, this is also possible.  In this case, the
"basic-auth": and filepath to the credentials file is passed as a pvo in the
"ws-protocols" section of the vhost definition.

@section lwswscc Requiring a Client Cert on a vhost

You can make a vhost insist to get a client certificate from the peer before
allowing the connection with

```
	"client-cert-required": "1"
```

the connection will only proceed if the client certificate was signed by the
same CA as the server has been told to trust.

@section rawconf Configuring Fallback and Raw vhosts

Lws supports some unusual modes for vhost listen sockets, which may be
configured entirely using the JSON per-vhost config language in the related
vhost configuration section.

There are three main uses for them

1) A vhost bound to a specific role and protocol, not http.  This binds all
incoming connections on the vhost listen socket to the "raw-proxy" role and
protocol "myprotocol".

```
	"listen-accept-role":		"raw-proxy",
	"listen-accept-protocol":	"myprotocol",
	"apply-listen-accept":		"1"
```

2) A vhost that wants to treat noncompliant connections for http or https as
   belonging to a secondary fallback role and protocol.  This causes non-https
   connections to an https listener to stop being treated as https, to lose the
   tls wrapper, and bind to role "raw-proxy" and protocol "myprotocol".  For
   example, connect a browser on your external IP :443 as usual and it serves
   as normal, but if you have configured the raw-proxy to portforward
   127.0.0.1:22, then connecting your ssh client to your external port 443 will
   instead proxy your sshd over :443 with no http or tls getting in the way.

```
	"listen-accept-role":		"raw-proxy",
	"listen-accept-protocol":	"myprotocol",
	"fallback-listen-accept":	"1",
	"allow-non-tls":		"1"
```

3) A vhost wants to either redirect stray http traffic back to https, or to
   actually serve http on an https listen socket (this is not recommended
   since it allows anyone to drop the security assurances of https by
   accident or design).

```
	"allow-non-tls":		"1",
	"redirect-http":		"1",
```

...or,

```
	"allow-non-tls":		"1",
	"allow-http-on-https":		"1",
```

@section lwswspl Lwsws Plugins

Protcols and extensions may also be provided from "plugins", these are
lightweight dynamic libraries.  They are scanned for at init time, and
any protocols and extensions found are added to the list given at context
creation time.

Protocols receive init (LWS_CALLBACK_PROTOCOL_INIT) and destruction
(LWS_CALLBACK_PROTOCOL_DESTROY) callbacks per-vhost, and there are arrangements
they can make per-vhost allocations and get hold of the correct pointer from
the wsi at the callback.

This allows a protocol to choose to strictly segregate data on a per-vhost
basis, and also allows the plugin to handle its own initialization and
context storage.

To help that happen conveniently, there are some new apis

 - lws_vhost_get(wsi)
 - lws_protocol_get(wsi)
 - lws_callback_on_writable_all_protocol_vhost(vhost, protocol)
 - lws_protocol_vh_priv_zalloc(vhost, protocol, size)
 - lws_protocol_vh_priv_get(vhost, protocol)

dumb increment, mirror and status protocol plugins are provided as examples.


@section lwswsplaplp Additional plugin search paths

Packages that have their own lws plugins can install them in their own
preferred dir and ask lwsws to scan there by using a config fragment
like this, in its own conf.d/ file managed by the other package
```
	{
	  "global": {
	   "plugin-dir": "/usr/local/share/coherent-timeline/plugins"
	  }
	}
```

@section lwswsssp lws-server-status plugin

One provided protocol can be used to monitor the server status.

Enable the protocol like this on a vhost's ws-protocols section
```
	       "lws-server-status": {
	         "status": "ok",
	         "update-ms": "5000"
	       }
```
`"update-ms"` is used to control how often updated JSON is sent on a ws link.

And map the provided HTML into the vhost in the mounts section
```
	       {
	        "mountpoint": "/server-status",
	        "origin": "file:///usr/local/share/libwebsockets-test-server/server-status",
	        "default": "server-status.html"
	       }
```
You might choose to put it on its own vhost which has "interface": "lo", so it's not
externally visible, or use the Basic Auth support to require authentication to
access it.

`"hide-vhosts": "{0 | 1}"` lets you control if information about your vhosts is included.
Since this includes mounts, you might not want to leak that information, mount names,
etc.

`"filespath":"{path}"` lets you give a server filepath which is read and sent to the browser
on each refresh.  For example, you can provide server temperature information on most
Linux systems by giving an appropriate path down /sys.

This may be given multiple times.


@section lwswsreload Lwsws Configuration Reload

You may send lwsws a `HUP` signal, by, eg

```
$ sudo killall -HUP lwsws
```

This causes lwsws to "deprecate" the existing lwsws process, and remove and close all of
its listen sockets, but otherwise allowing it to continue to run, until all
of its open connections close.

When a deprecated lwsws process has no open connections left, it is destroyed
automatically.

After sending the SIGHUP to the main lwsws process, a new lwsws process, which can
pick up the newly-available listen sockets, and use the current configuration
files, is automatically started.

The new configuration may differ from the original one in arbitrary ways, the new
context is created from scratch each time without reference to the original one.

Notes

1) Protocols that provide a "shared world" like mirror will have as many "worlds"
as there are lwsws processes still active.  People connected to a deprecated lwsws
process remain connected to the existing peers.

But any new connections will apply to the new lwsws process, which does not share
per-vhost "shared world" data with the deprecated process.  That means no new
connections on the deprecated context, ie a "shrinking world" for those guys, and a
"growing world" for people who connect after the SIGHUP.

2) The new lwsws process owes nothing to the previous one.  It starts with fresh
plugins, fresh configuration, fresh root privileges if that how you start it.

The plugins may have been updated in arbitrary ways including struct size changes
etc, and lwsws or lws may also have been updated arbitrarily.

3) A root parent process is left up that is not able to do anything except
respond to SIGHUP or SIGTERM.  Actual serving and network listening etc happens
in child processes which use the privileges set in the lwsws config files.

@section lwswssysd Lwsws Integration with Systemd

lwsws needs a service file like this as `/usr/lib/systemd/system/lwsws.service`
```
[Unit]
Description=Libwebsockets Web Server
After=syslog.target

[Service]
ExecStart=/usr/local/bin/lwsws
ExecReload=/usr/bin/killall -s SIGHUP lwsws ; sleep 1 ; /usr/local/bin/lwsws
StandardError=null

[Install]
WantedBy=multi-user.target
```

You can find this prepared in `./lwsws/usr-lib-systemd-system-lwsws.service`


@section lwswslr Lwsws Integration with logrotate

For correct operation with logrotate, `/etc/logrotate.d/lwsws` (if that's
where we're putting the logs) should contain
```
	/var/log/lwsws/*log {
	    copytruncate
	    missingok
	    notifempty
	    delaycompress
	}
```
You can find this prepared in `/lwsws/etc-logrotate.d-lwsws`

Prepare the log directory like this

```
	sudo mkdir /var/log/lwsws
	sudo chmod 700 /var/log/lwsws
```

@section lwswsgdb Debugging lwsws with gdb

Hopefully you won't need to debug lwsws itself, but you may want to debug your plugins.  start lwsws like this to have everything running under gdb

```
sudo gdb -ex "set follow-fork-mode child" -ex "run" --args /usr/local/bin/lwsws

```

this will give nice backtraces in lwsws itself and in plugins, if they were built with symbols.

@section lwswsvgd Running lwsws under valgrind

You can just run lwsws under valgrind as usual and get valid results.  However the results / analysis part of valgrind runs
after the plugins have removed themselves, this means valgrind backtraces into plugin code is opaque, without
source-level info because the dynamic library is gone.

There's a simple workaround, use LD_PRELOAD=<plugin.so> before running lwsws, this has the loader bring the plugin
in before executing lwsws as if it was a direct dependency.  That means it's still mapped until the whole process
exits after valgtind has done its thing.

lws-acme-client Plugin
======================

## Introduction

lws-acme-client is a protcol plugin for libwebsockets that implements an
ACME client able to communicate with let's encrypt and other certificate
providers.

It implements `tls-sni-01` challenge, and is able to provision tls certificates
"from thin air" that are accepted by all the major browsers.  It also manages
re-requesting the certificate when it only has two weeks left to run.

It works with both the OpenSSL and mbedTLS backends.

## Overview for use

You need to:

 - Provide name resolution to the IP with your server, ie, myserver.com needs to
 resolve to the IP that hosts your server

 - Enable port forwarding / external firewall access to your port, usually 443

 - Enable the "lws-acme-client" plugin on the vhosts you want it to manage
   certs for

 - Add per-vhost options describing what should be in the certificate

After that the plugin will sort everything else out.

## Example lwsws setup

```
 "vhosts": [ {
	"name": 		   "home.warmcat.com",
	"port":			   "443",
        "host-ssl-cert":           "/etc/lwsws/acme/home.warmcat.com.crt.pem",
        "host-ssl-key":            "/etc/lwsws/acme/home.warmcat.com.key.pem",
        "ignore-missing-cert":     "1",
	"access-log": 		   "/var/log/lwsws/test-access-log",
        "ws-protocols": [{
	  "lws-acme-client": {
	    "auth-path":	   "/etc/lwsws/acme/auth.jwk",
	    "cert-path":           "/etc/lwsws/acme/home.warmcat.com.crt.pem",
	    "key-path":            "/etc/lwsws/acme/home.warmcat.com.key.pem",
	    "directory-url":       "https://acme-staging.api.letsencrypt.org/directory",
	    "country":             "TW",
	    "state":               "Taipei",
	    "locality":            "Xiaobitan",
	    "organization":        "Crash Barrier Ltd",
	    "common-name":         "home.warmcat.com",
	    "email":               "andy@warmcat.com"
	  },
	  ...
```

## Required PVOs

Notice that the `"host-ssl-cert"` and `"host-ssl-key"` entries have the same
meaning as usual, they point to your certificate and private key.  However
because the ACME plugin can provision these, you should also mark the vhost with
`"ignore-missing-cert" : "1"`, so lwsws will ignore what will initially be
missing certificate / keys on that vhost, and will set about creating the
necessary certs and keys instead of erroring out.

You must make sure the directories mentioned here exist, lws doesn't create them
for you.  They should be 0700 root:root, even if you drop lws privileges.

If you are implementing support in code, this corresponds to making sure the
vhost creating `info.options` has the `LWS_SERVER_OPTION_IGNORE_MISSING_CERT`
bit set.

Similarly, in code, the each of the per-vhost options shown above can be
provided in a linked-list of structs at vhost creation time.  See
`./test-apps/test-server-v2.0.c` for example code for providing pvos.

### auth-path

This is where the plugin will store the auth keys it generated.

### cert-path

Where the plugin will store the certificate file.  Should match `host-ssl-cert`
that the vhost wants to use.

The path should include at least one 0700 root:root directory.

### key-path

Where the plugin will store the certificate keys.  Again it should match
`host-ssl-key` the vhost is trying to use.

The path should include at least one 0700 root:root directory.

### directory-url

This defines the URL of the certification server you will get your
certificates from.  For let's encrypt, they have a "practice" one

 - `https://acme-staging.api.letsencrypt.org/directory`

and they have a "real" one

 - `https://acme-v01.api.letsencrypt.org/directory`

the main difference is the CA certificate for the real one is in most browsers
already, but the staging one's CA certificate isn't.  The staging server will
also let you abuse it more in terms of repeated testing etc.

It's recommended you confirm expected operation with the staging directory-url,
and then switch to the "real" URL.

### common-name

Your server DNS name, like "libwebsockets.org".  The remote ACME server will
use this to find your server to perform the SNI challenges.

### email

The contact email address for the certificate.

## Optional PVOs

These are not included in the cert by letsencrypt

### country

Two-letter country code for the certificate

### state

State "or province" for the certificate

### locality

Locality for the certificate

### organization

Your company name

## Security / Key storage considerations

The `lws-acme-client` plugin is able to provision and update your certificate
and keys in an entirely root-only storage environment, even though lws runs
as a different uid / gid with no privileges to access the storage dir.

It does this by opening and holding two WRONLY fds on "update paths" inside the
root directory structure for each cert and key it manages; these are the normal
cert and key paths with `.upd` appended.  If during the time the server is up
the certs become within two weeks of expiry, the `lws-acme-client` plugin will
negotiate new certs and write them to the file descriptors.

Next time the server starts, if it sees `.upd` cert and keys, it will back up
the old ones and copy them into place as the new ones, before dropping privs.

To also handle the long-uptime server case, lws will update the vhost with the
new certs using in-memory temporary copies of the cert and key after updating
the cert.

In this way the cert and key live in root-only storage but the vhost is kept up
to date dynamically with any cert changes as well.

## Multiple vhosts using same cert

In the case you have multiple vhosts using of the same cert, just attach
the `lws-acme-client` plugin to one instance.  When the cert updates, all the
vhosts are informed and vhosts using the same filepath to access the cert will
be able to update their cert.

## Implementation point

You will need to remove the auth keys when switching from OpenSSL to
mbedTLS.  They will be regenerated automatically.  It's the file at this
path:

```
"auth-path":	   "/etc/lwsws/acme/auth.jwk",
```

ssh-base Plugin
================

## Introduction

lws-ssh-base is a protcol plugin for libwebsockets that implements a
generic, abstract, ssh server.

 - very small footprint in code and memory, takes up small part of ESP32

 - written with security in mind: valgrind and Coverity -clean

 - binds to one or more vhosts, that controls listen port(s)

 - all IO and settings abstracted through a single "ops" struct from user code

 - each instance on a vhost has its own "ops" struct, defining server keys,
   auth method and functions to implement IO and other operations

 - The plugin has no built-in behaviours like check ~/.ssh/authorized_keys,
   treat auth usernames as system usernames, or spawn the user's shell.
   Everything potentially dangerous is left to the user ops code to decide
   how to handle.  It's NOT like sshd where running it implies it will accept
   existing keys for any system user, will spawn a shell, etc, unless you
   implement those parts in the ops callbacks.

 - The plugin requires extra code around it in the form of the ops struct
   handlers.  So it's role is something like an abstract base class for an ssh
   server.  All the crypto, protocol sequencing and state machine are inside,
   but all the IO except the network connection is outside.

 - Built as part of libwebsockets, like all plugins may be dynamically loaded
   at runtime or built statically.  Test app `libwebsockets-test-sshd` provided

 - Uses hash and RSA functions from either mbedTLS or OpenSSL automatically,
   according to which library libwebsockets was built for

To maintain its small size, it implements a single "best of breed" crypto for
the following functions:

|Function|Crypto|
|---|---|
|KEX|curve25519-sha256@libssh.org|
|Server host key|ssh-rsa (4096b)|
|Encryption|chacha20-poly1305@openssh.com|
|Compression|None|

## License

lws-ssh-base is Free Software, available under libwebsockets' MIT license.

The crypto parts are available elsewhere under a BSD license.  But for
simplicity the whole plugin is under MIT.

## Generating your own keys

```
 $ ssh-keygen -t rsa -b 4096 -f mykeys
```

will ask for a passphrase and generate the private key in `mykeys` and the
public key in `mykeys.pub`.  If you already have a suitable RSA key you use
with ssh, you can just use that directly.

lws installs a test keypair in /usr[/local]/share/libwebsockets-test-server
that the test apps will accept.

## Example code

1) There's a working example app `libwebsockets-test-sshd` included that
spawns a bash shell when an ssh client authenticates.  The username used on
the remote ssh has no meaning, it spawns the shell under the credentials of
"lws-test-sshd" was run under.  It accepts the lws ssh test key which is
installed into /usr[/local]/share/libwebsockets-test-server.

Start the server like this (it wants root only because the server key is stored
in /etc)

```
 $ sudo libwebsockets-test-sshd
```

Connect to it using the test private key like this

```
 $ ssh -p 2200 -i /usr/local/share/libwebsockets-test-server/lws-ssh-test-keys anyuser@127.0.0.1
```

2) There's also a working example plugin `lws-sshd-demo` that "subclasses" the
abstract `lws-ssh-base` plugin to make a protocol which can be used from,
eg, lwsws.  For an lwsws vhost that listens on port 2222 and responds with
the lws-sshd-demo ssh server, the related config is:

```
        {
                "name": "sshd",
                "port": "2222",
                "onlyraw": "1",
                "ws-protocols": [{
                        "lws-ssh-base": {
                                "status": "ok",
                                "ops-from": "lws-sshd-demo"
                        },
                        "lws-sshd-demo": {
                                "status": "ok",
                                "raw": "1"
                        }
                }]
        }
```



## Integration to other apps

### Step 0: Build and install libwebsockets

For the `libwebsockets-test-sshd` example, you will need CMake options
`LWS_WITH_CGI`, since it uses lws helpers to spawn a shell.

lws-ssh-base itself doesn't require CGI support in libwebsockets.

### Step 1: make the code available in your app

Include `lws-plugin-ssh-base` in your app, either as a runtime plugin or by using
the lws static include scheme.

To bring in the whole of the ssh-base plugin
into your app in one step, statically, just include
`plugins/ssh-base/include/lws-plugin-sshd-static-build-includes.h`, you can see
an example of this in `./test-apps/test-sshd.c`.

### Step 2: define your `struct lws_ssh_ops`

`plugins/ssh-base/include/lws-plugin-ssh.h` defines
`struct lws_ssh_ops` which is used for all customization and integration
of the plugin per vhost.  Eg,

```
static const struct lws_ssh_ops ssh_ops = {
	.channel_create			= ssh_ops_channel_create,
	.channel_destroy		= ssh_ops_channel_destroy,
	.tx_waiting			= ssh_ops_tx_waiting,
	.tx				= ssh_ops_tx,
	.rx				= ssh_ops_rx,
	.get_server_key			= ssh_ops_get_server_key,
	.set_server_key			= ssh_ops_set_server_key,
	.set_env			= ssh_ops_set_env,
	.pty_req			= ssh_ops_pty_req,
	.child_process_io		= ssh_ops_child_process_io,
	.child_process_terminated	= ssh_ops_child_process_terminated,
	.exec				= ssh_ops_exec,
	.shell				= ssh_ops_shell,
	.is_pubkey_authorized		= ssh_ops_is_pubkey_authorized,
	.banner				= ssh_ops_banner,
	.disconnect_reason		= ssh_ops_disconnect_reason,
	.server_string			= "SSH-2.0-Libwebsockets",
	.api_version			= 1,
};
```
The `ssh_ops_...()` functions are your implementations for the operations
needed by the plugin for your purposes.

### Step 3: enable `lws-ssh-base` protocol to a vhost and configure using pvo

A pointer to your struct lws_ssh_ops is passed into the vhost instance of the
protocol using per-vhost options

```
static const struct lws_protocol_vhost_options pvo_ssh_ops = {
	NULL,
	NULL,
	"ops",
	(void *)&ssh_ops
};

static const struct lws_protocol_vhost_options pvo_ssh = {
	NULL,
	&pvo_ssh_ops,
	"lws-sshd-base",
	"" /* ignored, just matches the protocol name above */
};

...
	info.port = 22;
	info.options = LWS_SERVER_OPTION_ONLY_RAW;
	info.vhost_name = "sshd";
	info.protocols = protocols_sshd;
	info.pvo = &pvo_ssh;

	vh_sshd = lws_create_vhost(context, &info);
```

There are two possible pvos supported, "ops", shown above, directly passes the
ops structure in using the value on the "ops" pvo.

To support other protocols that want to provide ops to lws-ssh-base themselves
for a particular vhost, you can also provide a pvo `"ops-from"` whose value is
the name of the protocol also enabled on this vhost, whose protocol ".user"
pointer points to the ops struct lws-ssh-base should use.

## Integration to other plugins

A worked example of using the abstract `lws-ssh-base` plugin from another
plugin that provides the ops struct is in `./plugins/protocol_lws_sshd_demo`.

The key points to note

 - the plugin sets the ops struct for the vhost instantiation of `lws-ssh-base`
 by passing a pointer to the ops struct in its `lws_protocols` struct `user`
 member.

 - the config for the vhost tells `lws-ssh-base` to pick up the ops struct
 pointer using an "ops-from" pvo that indicates the protocol name.

```
 			"lws-ssh-base": {
                                "status": "ok",
                                "ops-from": "lws-sshd-demo"
                        },
```

 - the config for the vhost tells lws this vhost only serves RAW (ie, no http)

```
         {
                "name": "sshd",
                "port": "2222",
                "onlyraw": "1",
                ...
```

 - the config for the vhost marks the protocol that uses `lws-ssh-base`, not
 `lws-ssh-base` itself, as the protocol to be served for raw connections

```
                        "lws-sshd-demo": {
                                "status": "ok",
                                "raw": "1"
                         ...
```

## Notes

You can have the vhost it binds to listen on a nonstandard port.  The ssh
commandline app cane be told to connect to a non-22 port with
`ssh -p portnum user@hostname`

# Guidance for porting to new platform

Where differences existed between the initial POSIX platform for lws and other
supported platforms like Windows, `lws_plat_...()` apis were added to move
handling to platform-specific code in `./lib/plat/`.

Depending o which platform is built, different platform-specific implementations
of these `lws_plat...()` apis get built.

## 1) Prepare the cmake cross-build file if necessary

CMake isolates its settings for cross-build into a separate file, which can be
used to different cmake projects for the same platform as well.

Find a similar examples already in `./contrib/cross-*` and copy and adapt it
as needed,

All settings related to toolchain should go in there.  For cross-toolchain,
the convention is to pass the path to its installed directory in `CROSS_PATH`
environment variable.

## 2) Copy the closest platform dir in ./lib/plat

Wholesale copy the closest existing platform dir to `/lib/plat/myplatform` and
rename the files.

Remove stuff specific to the original platform.

## 3) Add a flag in CMakeLists.txt

Cut and paste a flag to select your platform, preferably `LWS_PLAT_MYPLATFORM` or so

## 4) Add a section to force-select and deselect other cmake options based on platform flag

Some options on by default may not make sense on your platform, and others off
by default may be mandatory.  After the options() section in CMakeLists.txt, you
can use this kind of structure

```
	if (LWS_PLAT_MYPLATFORM)
		set(LWS_WITH_XXXX 0)
	endif()
```

to enforce implicit requirements of your platform.  Optional stuff should be set by
running cmake commandline as usual.

## 5) Add building your platform files into CMakeLists.txt

Add entries in CMakeLists.txt for building stuff in `./lib/plat/myplatform` when
`LWS_PLAT_MYPLATFORM` is enabled.

## 6) Adapt your copied ./lib/plat/myplatform/ files

You can now do test builds using the cross-build file, your platform flag in
cmake, and your copied ./lib/plat content... this last part since it was
copied from another platform will initially be a plentiful source of errors.

You can iteratively build and adapt the platform files.

Debugging problems
==================

Check it's still a problem with latest lws
------------------------------------------

Older versions of lws don't attract any new work after they are released
(see [the release policy](https://libwebsockets.org/git/libwebsockets/tree/READMEs/README.release-policy.md) for details);
for a while they will get backported bugfixes but that's it.

All new work and bugfixes happen on master branch.

Old, old versions may be convenient for you to use for some reason.  But unless
you pay for support or have contributed work to lws so we feel we owe you some
consideration, nobody else has any reason to particularly care about solving
issues on ancient versions.  Whereas if the problem exists on master, and can be
reproduced by developers, it usually gets attention, often immediately.

If the problem doesn't exist on master, you can either use master or check also
the -stable branch of the last released version to see if it was already solved
there.

Library is a component
----------------------

As a library, lws is always just a component in a bigger application.

When users have a problem involving lws, what is happening in the bigger
application is usually critical to understand what is going on (and where the
solution lies).  Sometimes access to the remote peer like server or client is also
necessary to provoke the symptom.  Sometimes, the problem is in lws, but
sometimes the problem is not in lws but in these other pieces.

Many users are able to share their sources, but others decide not to, for
presumed "commercial advantage" or whatever.  (In any event, it can be painful
looking through large chunks of someone else's sources for problems when that
is not the library author's responsibility.)

This makes answering questions like "what is wrong with my code I am not
going to show you?" or even "what is wrong with my code?" very difficult.

Even if it's clear there is a problem somewhere, it cannot be understood or
reproduced by anyone else if it needs user code that isn't provided.

The biggest question is, "is this an lws problem actually"?  To solve that
the best solution is to strip out all or as much user code as possible,
and see if the problem is still coming.


Use the test apps / minimal examples as sanity checks
-----------------------------------------------------

The test server and client, and any more specifically relevant minimal example
 are extremely useful for sanity checks and debugging guidance.

 - **test apps work on your platform**, then either
   - your user code is broken, align it to how the test apps work, or,
   - something from your code is required to show an lws problem, provide a
     minimal patch on a test app so it can be reproduced

 - **test apps break on your platform**, but work on, eg, x86_64, either
   - toolchain or platform-specific (eg, OS) issue, or
   - lws platform support issue

 - **test apps break everywhere**
   - sounds like lws problem, info to reproduce and / or a patch is appreciated

# lws release policy

## How should users consume lws?

The one definitively wrong way to consume lws (or anything else) is "import" some
version of it into your proprietary tree and think you will stick with that
forever, perhaps piling cryptic fixes or hacks on top until quite quickly,
nobody dare think about updating it.

The stable releases go on to a branch like v4.0-stable as described below, over
time these attract dozens or even hundreds of minor or major fix patches
backported from master.  So you should not consume tags like v4.0.0 but build
into your planning that you will need to follow v4.0-stable in order to stay on
top of known bugs.

And we only backport fixes to the last stable release, although we will make
exceptions for important fixes.  So after a while, trying to stick with one
old versions means nobody is providing security fixes on it any more.  So you
should build into your planning that you will follow lws release upgrades.

If you find problems and create fixes, please upstream them, simplifying your
life so you can just directly consume the upstream tree with no private changes.

## Master branch

Master branch is the default and all new work happens there.  It's unstable and
subject to history rewrites, patches moving about and being squashed etc.  In
terms of it working, it is subject to passing CI tests including a battery of
runtime tests, so if it is passing CI as it usually is then it's probably in
usable shape.  See "Why no history on master" below for why it's managed like
that.

![all work happens on master](../doc-assets/lws-relpol-1.svg)

If you have patches (you are a hero) they should be targeted at master.

To follow such a branch, `git pull` is the wrong tool... the starting point
of what you currently have may no longer exist remotely due to rearranging the
patches there.  Instead use a flow like this:

```
 $ git fetch https://libwebsockets.org/repo/libwebsockets +master:m && git reset --hard m
```

This fetches current remote master into local branch `m`, and then forces your
local checkout to exactly match `m`.  This replaces your checked-out tree
including any of your local changes, so stash those first, or use stgit or so
to pop them before updating your basis against lws master.

## Stable branches

Master is very useful for coordinating development, and integrating WIP,
but for distros or integration into large user projects some stability is often
more desirable than the latest development work.

Periodically, when master seems in good shape and various new developments seem
to be working, it's copied out into a versioned stable branch, like `v3.0-stable`.

![stable branches are copied from master](../doc-assets/lws-relpol-2.svg)

The initial copy is tagged with, eg, `v3.0.0`.

(At that time, master's logical version is set to "...99", eg, `v3.0.99` so
version comparisons show that version of master is "later" than any other
v3.0 version, which will never reach 99 point releases itself, but "earlier"
than, eg, v3.1.)

## Backport policy

Work continues on master, and as part of that usually bugs are reported and / or
fixes found that may apply not just to current master, but the version of master
that was copied to form the last -stable branch.

In that case, the patch may be backported to the last stable branch to also fix
the bug there.  In the case of refactors or major internal improvements, these
typically do not get backported.

This applies only to fixes and public API-neutral internal changes to lws... if
new features were backported or API changes allowed, then there would be
multiple apis under the same version name and library soname, which is
madness.

When new stable releases are made, the soname is bumped reflecting the API is
different than that of previous versions.

![backports from master to stable](../doc-assets/lws-relpol-3.svg)

If there is something you need in a later lws version that is not backported,
you need to either backport it yourself or use a later lws version.
Using a more recent version of lws (and contributing fixes and changes so you
yourself can get them easily as well as contributing for others) is almost
always the correct way.

## Stable point releases

Periodically fix patches pile up on the -stable branch and are tagged out as
"point releases".  So if the original stable release was "v3.0.0", the point
release may be "v3.0.1".

![point releases of stable](../doc-assets/lws-relpol-4.svg)

## Critical fixes

Sometimes a bug is found and fixed that had been hiding for a few versions.
If the bug has some security dimension or is otherwise important, we may
backport it to a few recent releases, not just the last one.  This is pretty
uncommon though.

![backport to multiple stable branches](../doc-assets/lws-relpol-5.svg)

## Why no history on master

Git is a wonderful tool that can be understood to have two main modes of operation,
merge and rebase that are mutually exclusive.  Most devs only use merge / pull,
but rebase / fetch is much more flexible.  Running master via rebases allows me to
dispense with feature branches during development and enables tracking multiple
in-progress patches in-tree by updating them in place.  If this doesn't make
sense or seems heretical to you, it's OK I don't need devsplain'ing about it,
just sit back and enjoy the clean, rapid development results.

Since stable branches don't allow new features, they are run as traditional trees
with a history, like a one-way pile of patches on top of the original release.  If
CI shows something is messed up with the latest patch, I will edit it in-place if
it has only been out for a few hours, but there is no re-ordering or other history
manipulation.

Overview of lws test apps
=========================

Are you building a client?  You just need to look at the test client
[libwebsockets-test-client](../test-apps/test-client.c).

If you are building a standalone server, there are three choices, in order of
preferability.

1) lwsws + protocol plugins

Lws provides a generic web server app that can be configured with JSON
config files.  https://libwebsockets.org itself uses this method.

With lwsws handling the serving part, you only need to write an lws protocol
plugin.  See [plugin-standalone](../plugin-standalone) for an example of how
to do that outside lws itself, using lws public apis.

 $ cmake .. -DLWS_WITH_LWSWS=1

See [README.lwsws.md](../READMEs/README.lwsws.md) for information on how to configure
lwsws.

NOTE this method implies libuv is used by lws, to provide crossplatform
implementations of timers, dynamic lib loading etc for plugins and lwsws.

2) Using plugins in code

This method lets you configure web serving in code, instead of using lwsws.

Plugins are still used, but you have a choice whether to dynamically load
them or statically include them.  In this example, they are dynamically
loaded.

 $ cmake .. -DLWS_WITH_PLUGINS=1

See, eg, the [test-server](../test-apps/test-server.c)

3) protocols in the server app

This is the original way lws implemented servers, plugins and libuv are not
required, but without plugins separating the protocol code directly, the
combined code is all squidged together and is much less maintainable.

This method is still supported in lws but all ongoing and future work is
being done in protocol plugins only.

You can simply include the plugin contents and have it buit statically into
your server, just define this before including the plugin source

```
#define LWS_PLUGIN_STATIC
```

This gets you most of the advantages without needing dynamic loading +
libuv.


Notes about lws test apps
=========================

@section tsb Testing server with a browser

If you run [libwebsockets-test-server](../test-apps/test-server.c) and point your browser
(eg, Chrome) to

	http://127.0.0.1:7681

It will fetch a script in the form of `test.html`, and then run the
script in there on the browser to open a websocket connection.
Incrementing numbers should appear in the browser display.

By default the test server logs to both stderr and syslog, you can control
what is logged using `-d <log level>`, see later.


@section tsd Running test server as a Daemon

You can use the -D option on the test server to have it fork into the
background and return immediately.  In this daemonized mode all stderr is
disabled and logging goes only to syslog, eg, `/var/log/messages` or similar.

The server maintains a lockfile at `/tmp/.lwsts-lock` that contains the pid
of the master process, and deletes this file when the master process
terminates.

To stop the daemon, do
```
       $ kill \`cat /tmp/.lwsts-lock\`
```
If it finds a stale lock (the pid mentioned in the file does not exist
any more) it will delete the lock and create a new one during startup.

If the lock is valid, the daemon will exit with a note on stderr that
it was already running.

@section clicert Testing Client Certs

Here is a very quick way to create a CA, and a client and server cert from it,
for testing.

```
$ cp -rp ./scripts/client-ca /tmp
$ cd /tmp/client-ca
$ ./create-ca.sh
$ ./create-server-cert.sh server
$ ./create-client-cert.sh client
```

The last step wants an export password, you will need this password again to
import the p12 format certificate into your browser.

This will get you the following

|name|function|
|----|--------|
|ca.pem|Your Certificate Authority cert|
|ca.key|Private key for the CA cert|
|client.pem|Client certificate, signed by your CA|
|client.key|Client private key|
|client.p12|combined client.pem + client.key in p12 format for browsers|
|server.pem|Server cert, signed by your CA|
|server.key|Server private key|

You can confirm yourself the client and server certs are signed by the CA.

```
 $ openssl verify -verbose -trusted ca.pem server.pem
 $ openssl verify -verbose -trusted ca.pem client.pem
```

Import the client.p12 file into your browser.  In FFOX57 it's

 - preferences
 - Privacy & Security
 - Certificates | View Certificates
 - Certificate Manager | Your Certificates | Import...
 - Enter the password you gave when creating client1.p12
 - Click OK.

You can then run the test server like this:

```
 $ libwebsockets-test-server -s -A ca.pem -K server.key -C server.pem -v
```

When you connect your browser to https://localhost:7681 after accepting the
selfsigned server cert, your browser will pop up a prompt to send the server
your client cert (the -v switch enables this).  The server will only accept
a client cert that has been signed by ca.pem.

@section sssl Using SSL on the server side

To test it using SSL/WSS, just run the test server with
```
	$ libwebsockets-test-server --ssl
```
and use the URL
```
	https://127.0.0.1:7681
```
The connection will be entirely encrypted using some generated
certificates that your browser will not accept, since they are
not signed by any real Certificate Authority.  Just accept the
certificates in the browser and the connection will proceed
in first https and then websocket wss, acting exactly the
same.

[test-server.c](../test-apps/test-server.c) is all that is needed to use libwebsockets for
serving both the script html over http and websockets.

@section lwstsdynvhost Dynamic Vhosts

You can send libwebsockets-test-server or libwebsockets-test-server-v2.0 a SIGUSR1
to toggle the creation and destruction of an identical second vhost on port + 1.

This is intended as a test and demonstration for how to bring up and remove
vhosts dynamically.

@section unixskt Testing Unix Socket Server support

Start the test server with -U and the path to create the unix domain socket

```
 $ libwebsockets-test-server -U /tmp/uds
```

On exit, lws will delete the socket inode.

To test the client side, eg

```
 $ nc -C -U /tmp/uds -i 30
```

and type

`GET / HTTP/1.1`

followed by two ENTER.  The contents of test.html should be returned.

@section wscl Testing websocket client support

If you run the test server as described above, you can also
connect to it using the test client as well as a browser.

```
	$ libwebsockets-test-client localhost
```

will by default connect to the test server on localhost:7681
and print the dumb increment number from the server at the
same time as drawing random circles in the mirror protocol;
if you connect to the test server using a browser at the
same time you will be able to see the circles being drawn.

The test client supports SSL too, use

```
	$ libwebsockets-test-client localhost --ssl -s
```

the -s tells it to accept the default self-signed cert from the server,
otherwise it will strictly fail the connection if there is no CA cert to
validate the server's certificate.


@section choosingts Choosing between test server variations

If you will be doing standalone serving with lws, ideally you should avoid
making your own server at all, and use lwsws with your own protocol plugins.

The second best option is follow test-server-v2.0.c, which uses a mount to
autoserve a directory, and lws protocol plugins for ws, without needing any
user callback code (other than what's needed in the protocol plugin).

For those two options libuv is needed to support the protocol plugins, if
that's not possible then the other variations with their own protocol code
should be considered.

@section tassl Testing SSL on the client side

To test SSL/WSS client action, just run the client test with
```
	$ libwebsockets-test-client localhost --ssl
```
By default the client test applet is set to accept self-signed
certificates used by the test server, this is indicated by the
`use_ssl` var being set to `2`.  Set it to `1` to reject any server
certificate that it doesn't have a trusted CA cert for.


@section taping Using the websocket ping utility

libwebsockets-test-ping connects as a client to a remote
websocket server and pings it like the
normal unix ping utility.
```
	$ libwebsockets-test-ping localhost
	handshake OK for protocol lws-mirror-protocol
	Websocket PING localhost.localdomain (127.0.0.1) 64 bytes of data.
	64 bytes from localhost: req=1 time=0.1ms
	64 bytes from localhost: req=2 time=0.1ms
	64 bytes from localhost: req=3 time=0.1ms
	64 bytes from localhost: req=4 time=0.2ms
	64 bytes from localhost: req=5 time=0.1ms
	64 bytes from localhost: req=6 time=0.2ms
	64 bytes from localhost: req=7 time=0.2ms
	64 bytes from localhost: req=8 time=0.1ms
	^C
	--- localhost.localdomain websocket ping statistics ---
	8 packets transmitted, 8 received, 0% packet loss, time 7458ms
	rtt min/avg/max = 0.110/0.185/0.218 ms
	$
```
By default it sends 64 byte payload packets using the 04
PING packet opcode type.  You can change the payload size
using the `-s=` flag, up to a maximum of 125 mandated by the
04 standard.

Using the lws-mirror protocol that is provided by the test
server, libwebsockets-test-ping can also use larger payload
sizes up to 4096 is BINARY packets; lws-mirror will copy
them back to the client and they appear as a PONG.  Use the
`-m` flag to select this operation.

The default interval between pings is 1s, you can use the -i=
flag to set this, including fractions like `-i=0.01` for 10ms
interval.

Before you can even use the PING opcode that is part of the
standard, you must complete a handshake with a specified
protocol.  By default lws-mirror-protocol is used which is
supported by the test server.  But if you are using it on
another server, you can specify the protocol to handshake with
by `--protocol=protocolname`


@section ta fraggle Fraggle test app

By default it runs in server mode
```
	$ libwebsockets-test-fraggle
	libwebsockets test fraggle
	(C) Copyright 2010-2011 Andy Green <andy@warmcat.com> licensed under MIT
	 Compiled with SSL support, not using it
	 Listening on port 7681
	server sees client connect
	accepted v06 connection
	Spamming 360 random fragments
	Spamming session over, len = 371913. sum = 0x2D3C0AE
	Spamming 895 random fragments
	Spamming session over, len = 875970. sum = 0x6A74DA1
	...
```
You need to run a second session in client mode, you have to
give the `-c` switch and the server address at least:
```
	$ libwebsockets-test-fraggle -c localhost
	libwebsockets test fraggle
	(C) Copyright 2010-2011 Andy Green <andy@warmcat.com> licensed under MIT
	 Client mode
	Connecting to localhost:7681
	denied deflate-stream extension
	handshake OK for protocol fraggle-protocol
	client connects to server
	EOM received 371913 correctly from 360 fragments
	EOM received 875970 correctly from 895 fragments
	EOM received 247140 correctly from 258 fragments
	EOM received 695451 correctly from 692 fragments
	...
```
The fraggle test sends a random number up to 1024 fragmented websocket frames
each of a random size between 1 and 2001 bytes in a single message, then sends
a checksum and starts sending a new randomly sized and fragmented message.

The fraggle test client receives the same message fragments and computes the
same checksum using websocket framing to see when the message has ended.  It
then accepts the server checksum message and compares that to its checksum.


@section taproxy proxy support

The http_proxy environment variable is respected by the client
connection code for both `ws://` and `wss://`.  It doesn't support
authentication.

You use it like this
```
	$ export http_proxy=myproxy.com:3128
	$ libwebsockets-test-client someserver.com
```

@section talog debug logging

By default logging of severity "notice", "warn" or "err" is enabled to stderr.

Again by default other logging is compiled in but disabled from printing.

By default debug logs below "notice" in severity are not compiled in.  To get
them included, add this option in CMAKE

```
	$ cmake .. -DCMAKE_BUILD_TYPE=DEBUG
```

If you want to see more detailed debug logs, you can control a bitfield to
select which logs types may print using the `lws_set_log_level()` api, in the
test apps you can use `-d <number>` to control this.  The types of logging
available are (OR together the numbers to select multiple)

 - 1   ERR
 - 2   WARN
 - 4   NOTICE
 - 8   INFO
 - 16  DEBUG
 - 32  PARSER
 - 64  HEADER
 - 128 EXTENSION
 - 256 CLIENT
 - 512 LATENCY


@section ws13 Websocket version supported

The final IETF standard is supported for both client and server, protocol
version 13.


@section latency Latency Tracking

Since libwebsockets runs using `poll()` and a single threaded approach, any
unexpected latency coming from system calls would be bad news.  There's now
a latency tracking scheme that can be built in with `-DLWS_WITH_LATENCY=1` at
cmake, logging the time taken for system calls to complete and if
the whole action did complete that time or was deferred.

You can see the detailed data by enabling logging level 512 (eg, `-d 519` on
the test server to see that and the usual logs), however even without that
the "worst" latency is kept and reported to the logs with NOTICE severity
when the context is destroyed.

Some care is needed interpreting them, if the action completed the first figure
(in us) is the time taken for the whole action, which may have retried through
the poll loop many times and will depend on network roundtrip times.  High
figures here don't indicate a problem.  The figure in us reported after "lat"
in the logging is the time taken by this particular attempt.  High figures
here may indicate a problem, or if you system is loaded with another app at
that time, such as the browser, it may simply indicate the OS gave preferential
treatment to the other app during that call.


@section autobahn Autobahn Test Suite

Lws can be tested against the autobahn websocket fuzzer in both client and
server modes

1) pip install autobahntestsuite

2) From your build dir:

```
 $ cmake .. -DLWS_WITHOUT_EXTENSIONS=0 -DLWS_WITH_MINIMAL_EXAMPLES=1 && make
```

3) ../scripts/autobahn-test.sh

4) In a browser go to the directory you ran wstest in (eg, /projects/libwebsockets)

file:///projects/libwebsockets/build/reports/clients/index.html

to see the results


@section autobahnnotes Autobahn Test Notes

1) Two of the tests make no sense for Libwebsockets to support and we fail them.

 - Tests 2.10 + 2.11: sends multiple pings on one connection.  Lws policy is to
only allow one active ping in flight on each connection, the rest are dropped.
The autobahn test itself admits this is not part of the standard, just someone's
random opinion about how they think a ws server should act.  So we will fail
this by design and it is no problem about RFC6455 compliance.

2) Currently two parts of autobahn are broken and we skip them

https://github.com/crossbario/autobahn-testsuite/issues/71

## Using UDP in lws

UDP is supported in lws... the quickest way is to use the api
`lws_create_adopt_udp()` which returns a wsi bound to the provided
vhost, protocol, `lws_retry` struct, dns address and port.

The wsi can be treated normally and `lws_write()` used to write on
it.

## Implementing UDP retries

Retries are important in udp but there's no standardized ack method
unlike tcp.  Lws allows you to bind an `lws_retry` struct describing
the policy to the udp wsi, but since one UDP socket may have many
transactions in flight, the `lws_sul` and `uint16_t` to count the
retries must live in the user's transaction object like this

```
...
	lws_sorted_usec_list_t	sul;
	uint16_t		retry;
...
```

in the `LWS_CALLBACK_RAW_WRITEABLE` callback, before doing the write,
set up the retry like this

```
	if (lws_dll2_is_detached(&transaction->sul_write.list) &&
	    lws_retry_sul_schedule_retry_wsi(wsi, &transaction->sul_write,
					     transaction_retry_write_cb,
					     &transaction->retry_count_write)) {
			/* we have reached the end of our concealed retries */
		lwsl_warn("%s: concealed retries done, failing\n", __func__);
		goto retry_conn;
	}
```

This manages the retry counter in the transaction object, guards against it wrapping,
selects the timeout using the policy bound to the wsi, and sets the `lws_sul` in the
transaction object to call the given callback if the sul time expires.

In the callback, it should simply call `lws_callback_on_writable()` for the udp wsi.

## Simulating packetloss

lws now allows you to set the amount of simulated packetloss on udp rx and tx in
the context creation info struct, using `.udp_loss_sim_tx_pc` and `.udp_loss_sim_rx_pc`,
the values are percentages between 0 and 100.  0, the default, means no packetloss.

## Unix Domain Sockets Reverse Proxy

### Introduction

lws is able to use a mount to place reverse proxies into the URL space.

These are particularly useful when using Unix Domain Sockets, basically
files in the server filesystem, to communicate between lws and a separate
server process and integrate the result into a coherent URL namespace on
the lws side.  It's also possible to proxy using tcp sockets.

![overview](../doc-assets/http-proxy-overview.svg)

This has the advantage that the actual web server that forwards the
data from the unix socket owner is in a different process than the server
that serves on the unix socket.  If it has problems, they do not affect
the actual public-facing web server.  The unix domain socket server may
be in a completely different language than the web server.

Compared to CGI, there are no forks to make a connection to the unix
domain socket server.

### Mount origin format

Unix Domain Sockets are effectively "files" in the server filesystem, and
are defined by their filepath.  The "server" side that is to be proxied opens
the socket and listens on it, which creates a file in the server filesystem.
The socket understands either http or https protocol.

Lws can be told to act as a proxy for that at a mountpoint in the lws vhost
url space.

If your mount is expressed in C code, then the mount type is LWSMPRO_HTTP or
LWSMPRO_HTTPS depending on the protocol the unix socket understands, and the
origin address has the form `+/path/to/unix/socket:/path/inside/mount`.

The + at the start indicates it is a local unix socket we are proxying, and
the ':' acts as a delimiter for the socket path, since unlike other addresses
the unix socket path can contain '/' itself.

### Connectivity rules and translations

Onward proxy connections from lws to the Unix Domain Socket happen using
http/1.1.  That implies `transfer-encoding: chunking` in the case that the
length of the output is not known beforehand.

Lws takes care of stripping any chunking (which is illegal in h2) and
translating between h1 and h2 header formats if the return connection is
actually in http/2.

The h1 onward proxy connection translates the following headers from the return
connection, which may be h1 or h2:

Header|Function
---|---
host|Which vhost
etag|Information on any etag the client has cached for this URI
if-modified-since|Information on the freshness of any etag the client has cached for this URI
accept-language|Which languages the return path client prefers
accept-encoding|Which compression encodings the client can accept
cache-control|Information from the return path client about cache acceptability
x-forwarded-for|The IP address of the return path client

This implies that the proxied connection can

 - return 301 etc to say the return path client's etag is still valid

 - choose to compress using an acceptable content-encoding

The following headers are translated from the headers replied via the onward
connection (always h1) back to the return path (which may be h1 or h2)

Header|Function
---|---
content-length|If present, an assertion of how much payload is expected
content-type|The mimetype of the payload
etag|The canonical etag for the content at this URI
accept-language|This is returned to the return path client because there is no easy way for the return path client to know what it sent originally.  It allows clientside selection of i18n.
content-encoding|Any compression format on the payload (selected from what the client sent in accept-encoding, if anything)
cache-control|The onward server's response about cacheability of its payload

### h1 -> h2 conversion

Chunked encoding that may have been used on the outgoing proxy client connection
is removed for h2 return connections (chunked encoding is illegal for h2).

Headers are converted to all lower-case and hpack format for h2 return connections.

Header and payload proxying is staged according to when the return connection
(which may be an h2 child stream) is writable.

### Behaviour if unix domain socket server unavailable

If the server that listens on the unix domain socket is down or being restarted,
lws understands that it couldn't connect to it and returns a clean 503 response
`HTTP_STATUS_SERVICE_UNAVAILABLE` along with a brief human-readable explanation.

The generated status page produced will try to bring in a stylesheet
`/error.css`.  This allows you to produce a styled error pages with logos,
graphics etc.  See [this](https://libwebsockets.org/git/badrepo) for an example of what you can do with it.

## Vulnerability Reporting

If you become aware of an issue with lws that has a security
dimension for users, please contact `andy@warmcat.com` by
direct email.

## Procedure for announcing vulnerability fixes

The problem and fixed versions will be announced on the
libwebsockets mailing list and a note added to the master
README.md.