• Home
  • History
  • Annotate
Name Date Size #Lines LOC

..--

src/23-Nov-2023-19,78111,601

tests/23-Nov-2023-2,4241,677

.cargo_vcs_info.jsonD23-Nov-202374 65

.gitignoreD23-Nov-2023107 109

.travis.ymlD23-Nov-20232.3 KiB10287

Android.bpD23-Nov-2023855 4136

CHANGELOG.mdD23-Nov-202348.2 KiB1,171876

Cargo.tomlD23-Nov-20233.4 KiB182145

Cargo.toml.origD23-Nov-20232.8 KiB182141

LICENSED23-Nov-20231 KiB2117

METADATAD23-Nov-2023375 2019

MODULE_LICENSE_MITD23-Nov-20230

OWNERSD23-Nov-202340 21

README.mdD23-Nov-202312.5 KiB301233

TEST_MAPPINGD23-Nov-2023145 98

build.rsD23-Nov-2023155 86

README.md

1# nom, eating data byte by byte
2
3[![LICENSE](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
4[![Join the chat at https://gitter.im/Geal/nom](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/Geal/nom?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
5[![Build Status](https://travis-ci.org/Geal/nom.svg?branch=master)](https://travis-ci.org/Geal/nom)
6[![Coverage Status](https://coveralls.io/repos/Geal/nom/badge.svg?branch=master)](https://coveralls.io/r/Geal/nom?branch=master)
7[![Crates.io Version](https://img.shields.io/crates/v/nom.svg)](https://crates.io/crates/nom)
8[![Minimum rustc version](https://img.shields.io/badge/rustc-1.31.0+-lightgray.svg)](#rust-version-requirements)
9
10nom is a parser combinators library written in Rust. Its goal is to provide tools
11to build safe parsers without compromising the speed or memory consumption. To
12that end, it uses extensively Rust's *strong typing* and *memory safety* to produce
13fast and correct parsers, and provides functions, macros and traits to abstract most of the
14error prone plumbing.
15
16![nom logo in CC0 license, by Ange Albertini](https://raw.githubusercontent.com/Geal/nom/master/assets/nom.png)
17
18*nom will happily take a byte out of your files :)*
19
20## Example
21
22[Hexadecimal color](https://developer.mozilla.org/en-US/docs/Web/CSS/color) parser:
23
24```rust
25extern crate nom;
26use nom::{
27  IResult,
28  bytes::complete::{tag, take_while_m_n},
29  combinator::map_res,
30  sequence::tuple
31};
32
33#[derive(Debug,PartialEq)]
34pub struct Color {
35  pub red:   u8,
36  pub green: u8,
37  pub blue:  u8,
38}
39
40fn from_hex(input: &str) -> Result<u8, std::num::ParseIntError> {
41  u8::from_str_radix(input, 16)
42}
43
44fn is_hex_digit(c: char) -> bool {
45  c.is_digit(16)
46}
47
48fn hex_primary(input: &str) -> IResult<&str, u8> {
49  map_res(
50    take_while_m_n(2, 2, is_hex_digit),
51    from_hex
52  )(input)
53}
54
55fn hex_color(input: &str) -> IResult<&str, Color> {
56  let (input, _) = tag("#")(input)?;
57  let (input, (red, green, blue)) = tuple((hex_primary, hex_primary, hex_primary))(input)?;
58
59  Ok((input, Color { red, green, blue }))
60}
61
62fn main() {}
63
64#[test]
65fn parse_color() {
66  assert_eq!(hex_color("#2F14DF"), Ok(("", Color {
67    red: 47,
68    green: 20,
69    blue: 223,
70  })));
71}
72```
73
74## Documentation
75
76- [Reference documentation](https://docs.rs/nom)
77- [Various design documents and tutorials](https://github.com/Geal/nom/tree/master/doc)
78- [list of combinators and their behaviour](https://github.com/Geal/nom/blob/master/doc/choosing_a_combinator.md)
79
80If you need any help developing your parsers, please ping `geal` on IRC (freenode, geeknode, oftc), go to `#nom-parsers` on Freenode IRC, or on the [Gitter chat room](https://gitter.im/Geal/nom).
81
82## Why use nom
83
84If you want to write:
85
86### binary format parsers
87
88nom was designed to properly parse binary formats from the beginning. Compared
89to the usual handwritten C parsers, nom parsers are just as fast, free from
90buffer overflow vulnerabilities, and handle common patterns for you:
91
92- [TLV](https://en.wikipedia.org/wiki/Type-length-value)
93- bit level parsing
94- hexadecimal viewer in the debugging macros for easy data analysis
95- streaming parsers for network formats and huge files
96
97Example projects:
98
99- [FLV parser](https://github.com/rust-av/flavors)
100- [Matroska parser](https://github.com/rust-av/matroska)
101- [tar parser](https://github.com/Keruspe/tar-parser.rs)
102
103### Text format parsers
104
105While nom was made for binary format at first, it soon grew to work just as
106well with text formats. From line based formats like CSV, to more complex, nested
107formats such as JSON, nom can manage it, and provides you with useful tools:
108
109- fast case insensitive comparison
110- recognizers for escaped strings
111- regular expressions can be embedded in nom parsers to represent complex character patterns succinctly
112- special care has been given to managing non ASCII characters properly
113
114Example projects:
115
116- [HTTP proxy](https://github.com/sozu-proxy/sozu/blob/master/lib/src/protocol/http/parser.rs)
117- [TOML parser](https://github.com/joelself/tomllib)
118
119### Programming language parsers
120
121While programming language parsers are usually written manually for more
122flexibility and performance, nom can be (and has been successfully) used
123as a prototyping parser for a language.
124
125nom will get you started quickly with powerful custom error types, that you
126can leverage with [nom_locate](https://github.com/fflorent/nom_locate) to
127pinpoint the exact line and column of the error. No need for separate
128tokenizing, lexing and parsing phases: nom can automatically handle whitespace
129parsing, and construct an AST in place.
130
131Example projects:
132
133- [PHP VM](https://github.com/tagua-vm/parser)
134- eve language prototype
135- [xshade shading language](https://github.com/xshade-lang/xshade/)
136
137### Streaming formats
138
139While a lot of formats (and the code handling them) assume that they can fit
140the complete data in memory, there are formats for which we only get a part
141of the data at once, like network formats, or huge files.
142nom has been designed for a correct behaviour with partial data: if there is
143not enough data to decide, nom will tell you it needs more instead of silently
144returning a wrong result. Whether your data comes entirely or in chunks, the
145result should be the same.
146
147It allows you to build powerful, deterministic state machines for your protocols.
148
149Example projects:
150
151- [HTTP proxy](https://github.com/sozu-proxy/sozu/blob/master/lib/src/protocol/http/parser.rs)
152- [using nom with generators](https://github.com/Geal/generator_nom)
153
154## Parser combinators
155
156Parser combinators are an approach to parsers that is very different from
157software like [lex](https://en.wikipedia.org/wiki/Lex_(software)) and
158[yacc](https://en.wikipedia.org/wiki/Yacc). Instead of writing the grammar
159in a separate file and generating the corresponding code, you use very
160small functions with very specific purpose, like "take 5 bytes", or
161"recognize the word 'HTTP'", and assemble then in meaningful patterns
162like "recognize 'HTTP', then a space, then a version".
163The resulting code is small, and looks like the grammar you would have
164written with other parser approaches.
165
166This has a few advantages:
167
168- the parsers are small and easy to write
169- the parsers components are easy to reuse (if they're general enough, please add them to nom!)
170- the parsers components are easy to test separately (unit tests and property-based tests)
171- the parser combination code looks close to the grammar you would have written
172- you can build partial parsers, specific to the data you need at the moment, and ignore the rest
173
174## Technical features
175
176nom parsers are for:
177- [x] **byte-oriented**: the basic type is `&[u8]` and parsers will work as much as possible on byte array slices (but are not limited to them)
178- [x] **bit-oriented**: nom can address a byte slice as a bit stream
179- [x] **string-oriented**: the same kind of combinators can apply on UTF-8 strings as well
180- [x] **zero-copy**: if a parser returns a subset of its input data, it will return a slice of that input, without copying
181- [x] **streaming**: nom can work on partial data and detect when it needs more data to produce a correct result
182- [x] **descriptive errors**: the parsers can aggregate a list of error codes with pointers to the incriminated input slice. Those error lists can be pattern matched to provide useful messages.
183- [x] **custom error types**: you can provide a specific type to improve errors returned by parsers
184- [x] **safe parsing**: nom leverages Rust's safe memory handling and powerful types, and parsers are routinely fuzzed and tested with real world data. So far, the only flaws found by fuzzing were in code written outside of nom
185- [x] **speed**: benchmarks have shown that nom parsers often outperform many parser combinators library like Parsec and attoparsec, some regular expression engines and even handwritten C parsers
186
187Some benchmarks are available on [Github](https://github.com/Geal/nom_benchmarks).
188
189## Rust version requirements
190
191The 5.0 series of nom requires **Rustc version 1.31 or greater**.
192
193Travis CI always has a build with a pinned version of Rustc matching the oldest supported Rust release.
194The current policy is that this will only be updated in the next major nom release.
195
196## Installation
197
198nom is available on [crates.io](https://crates.io/crates/nom) and can be included in your Cargo enabled project like this:
199
200```toml
201[dependencies]
202nom = "5"
203```
204
205Then include it in your code like this:
206
207```rust,ignore
208#[macro_use]
209extern crate nom;
210```
211
212**NOTE: if you have existing code using nom below the 5.0 version, please take a look
213at the [upgrade documentation](https://github.com/Geal/nom/blob/master/doc/upgrading_to_nom_5.md)
214to handle the breaking changes.**
215
216There are a few compilation features:
217
218* `std`: (activated by default) if disabled, nom can work in `no_std` builds
219* `regexp`: enables regular expression parsers with the `regex` crate
220* `regexp_macros`: enables regular expression parsers with the `regex` and `regex_macros` crates. Regular expressions can be defined at compile time, but it requires a nightly version of rustc
221
222You can activate those features like this:
223
224```toml
225[dependencies.nom]
226version = "^5"
227features = ["regexp"]
228```
229
230# Related projects
231
232- [get line and column info in nom's input type](https://github.com/fflorent/nom_locate)
233- [using nom as lexer and parser](https://github.com/Rydgel/monkey-rust)
234
235# Parsers written with nom
236
237Here is a (non exhaustive) list of known projects using nom:
238
239- Text file formats:
240  * [Ceph Crush](https://github.com/cholcombe973/crushtool)
241  * [Cronenberg](https://github.com/ayrat555/cronenberg)
242  * [XFS Runtime Stats](https://github.com/ChrisMacNaughton/xfs-rs)
243  * [CSV](https://github.com/GuillaumeGomez/csv-parser)
244  * [FASTQ](https://github.com/elij/fastq.rs)
245  * [INI](https://github.com/Geal/nom/blob/master/tests/ini.rs)
246  * [ISO 8601 dates](https://github.com/badboy/iso8601)
247  * [libconfig-like configuration file format](https://github.com/filipegoncalves/rust-config)
248  * [Web archive](https://github.com/sbeckeriv/warc_nom_parser)
249  * [proto files](https://github.com/tafia/protobuf-parser)
250  * [Fountain screenplay markup](https://github.com/adamchalmers/fountain-rs)
251- Programming languages:
252  * [PHP](https://github.com/tagua-vm/parser)
253  * [Basic Calculator](https://github.com/balajisivaraman/basic_calculator_rs)
254  * [GLSL](https://github.com/phaazon/glsl)
255  * [Lua](https://github.com/doomrobo/nom-lua53)
256  * [Python](https://github.com/ProgVal/rust-python-parser)
257  * [SQL](https://github.com/ms705/nom-sql)
258  * [Elm](https://github.com/cout970/Elm-interpreter)
259  * [SystemVerilog](https://github.com/dalance/sv-parser)
260  * [Turtle](https://github.com/vandenoever/rome/tree/master/src/io/turtle)
261  * [CSML](https://github.com/CSML-by-Clevy/csml-interpreter)
262- Interface definition formats:
263  * [Thrift](https://github.com/thehydroimpulse/thrust)
264- Audio, video and image formats:
265  * [GIF](https://github.com/Geal/gif.rs)
266  * [MagicaVoxel .vox](https://github.com/davidedmonds/dot_vox)
267  * [midi](https://github.com/derekdreery/nom-midi-rs)
268  * [SWF](https://github.com/open-flash/swf-parser)
269  * [WAVE](http://github.com/noise-Labs/wave)
270- Document formats:
271  * [TAR](https://github.com/Keruspe/tar-parser.rs)
272  * [GZ](https://github.com/nharward/nom-gzip)
273- Cryptographic formats:
274  * [X.509](https://github.com/rusticata/x509-parser)
275- Network protocol formats:
276  * [Bencode](https://github.com/jbaum98/bencode.rs)
277  * [DHCP](https://github.com/rusticata/dhcp-parser)
278  * [HTTP](https://github.com/sozu-proxy/sozu/tree/master/lib/src/protocol/http)
279  * [URI](https://github.com/santifa/rrp/blob/master/src/uri.rs)
280  * [IMAP](https://github.com/djc/tokio-imap)
281  * [IRC](https://github.com/Detegr/RBot-parser)
282  * [Pcap-NG](https://github.com/richo/pcapng-rs)
283  * [Pcap](https://github.com/ithinuel/pcap-rs)
284  * [Pcap + PcapNG](https://github.com/rusticata/pcap-parser)
285  * [IKEv2](https://github.com/rusticata/ipsec-parser)
286  * [NTP](https://github.com/rusticata/ntp-parser)
287  * [SNMP](https://github.com/rusticata/snmp-parser)
288  * [Kerberos v5](https://github.com/rusticata/kerberos-parser)
289  * [DER](https://github.com/rusticata/der-parser)
290  * [TLS](https://github.com/rusticata/tls-parser)
291  * [IPFIX / Netflow v10](https://github.com/dominotree/rs-ipfix)
292  * [GTP](https://github.com/fuerstenau/gorrosion-gtp)
293- Language specifications:
294  * [BNF](https://github.com/snewt/bnf)
295- Misc formats:
296  * [Gameboy ROM](https://github.com/MarkMcCaskey/gameboy-rom-parser)
297
298Want to create a new parser using `nom`? A list of not yet implemented formats is available [here](https://github.com/Geal/nom/issues/14).
299
300Want to add your parser here? Create a pull request for it!
301