10 steps to yaml zen
We all love Ansible, but Ansible is YAML. There are many formats for configuration files: lists of values, parameter-value pairs, INI files, YAML, JSON, XML, and many others. However, for several reasons, of all of these, YAML is often considered particularly difficult. In particular, despite its refreshing minimalism and impressive ability to work with hierarchical values, the YAML syntax can be annoying with its Python-like indentation approach.
If YAML infuriates you, you can - and must! - Take the following 10 steps to reduce your irritation to an acceptable level and love YAML. As befits this list, our ten tips will be numbered from scratch, we add meditation and spiritual practices at will ;-)
It doesn't matter what kind of text editor you have - for him, for sure, there is at least one plugin for working with YAML. If you don’t have one, find and install immediately. The time spent on searching and setting up will pay off many times each time you have to edit YAML.
For example, the Atom editor supports YAML by default, but GNU Emacs will have to install additional packages, for example, yaml-mode .
Emacs in YAML mode and display spaces.
If your favorite editor does not have YAML mode, then some of the problems can be solved by working with the settings. For example, the standard Gedit text editor for GNOME does not have YAML mode, but it defaults to highlighting YAML syntax and allows you to configure work with indentation:
Gedit indentation settings.
And the drawspaces plugin for Gedit displays spaces as dots, eliminating ambiguities with indentation levels.
In other words, take the time to study your favorite editor. Find out what he or his development community is offering to work with YAML, and use these features. You definitely will not regret it.
Ideally, programming and markup languages ​​use predictable syntax. Computers do a good job of predictability, so back in 1978 the concept of a linter appeared . If for 40 years of its existence it passed you and you still do not use the YAML-linter, then it's time to try yamllint.
You can install yamllint using the standard Linux package manager. For example, on Red Hat Enterprise Linux 8 or Fedora, this is done like this:
Then you just run yamllint, passing it the YAML file for verification. Here's what it looks like if you pass an error file to the linter:
The numbers on the left are not time, but the coordinates of the error: row and column number. The error description may not tell you anything, but you know exactly where it is. Just look at this place in the code, and most likely, everything will become clear.
When yamllint does not find errors in the file, nothing is displayed. If you are afraid of such silence and want a little more feedback, then you can start the linter with the conditional echo command through the double ampersand (&&), like this:
In POSIX, a double ampersand is triggered if and only if the previous command returns 0. And yamllint just returns the number of errors found, so this whole conditional construction works.
If YAML really pisses you off, just don't write on it, literally. It happens that YAML is the only format that is perceived by the application. But in this case, it is not necessary to create a YAML file. Write on what you like, and then convert. For example, for Python there is an excellent pyyaml library and two whole conversion methods: self-conversion and conversion through scripts.
In this case, the data file is also a Python script that generates YAML. This method is best suited for small data sets. You simply write JSON data to a Python variable, precede it with the import directive, and add three lines at the end of the file to implement the output.
Now run this file in Python and the output file is output.yaml:
This is absolutely correct YAML, but yamllint will warn you that it does not start with ---. Well, this can be easily fixed manually or slightly modified Python-script.
In this case, we first write in JSON and then run the converter in the form of a separate Python script, which yields YAML in the output. Compared to the previous one, this method scales better because the conversion is separated by data.
First, create the JSON file example.json, for example, you can take it on json.org :
Then create a simple script converter and save it under the name json2yaml.py. This script imports both modules - YAML and JSON Python, and also downloads the user specified JSON file, performs conversion and writes data to the output.yaml file.
Save this script in the system path and run as needed:
Sometimes it’s useful to look at the problem from a different angle. If you find it difficult to imagine the relationships between data in YAML, you can temporarily transform them into something more familiar.
For example, if you are comfortable working with dictionary lists or with JSON, then YAML can be converted to JSON with just two commands in the interactive Python shell. Suppose you have a mydata.yaml YAML file, then here is how it would look:
A lot of other examples can be found on this subject. In addition, there are many online converters and local parsers. So do not hesitate to reformat the data when you see in them only an incomprehensible hash.
Returning to YAML after a long break, it’s useful to go to yaml.org and re-read the specs. If you have difficulties with YAML, but your hands haven’t reached the specification, then it’s time to correct this situation. The specs are surprisingly easy to write, and the syntax requirements are illustrated by a large number of examples in Chapter 6 .
When writing a book or article, it is always useful to first draft a preliminary plan, at least in the form of a table of contents. So it is with YAML. Most likely, you can imagine what data you need to write to the YAML file, but you don’t really understand how to connect them to each other. Therefore, before sculpting YAML, draw a pseudo-config.
Pseudo-config is similar to pseudo-code, where you do not have to worry about structure or indentation, parent-child relationships, inheritance and nesting. So it is here: you draw iterations of data as they arise in your head.
A pseudo-config listing programmers (Martin and Tabitha) and their skills (programming languages: Python, Perl, Pascal and Lisp, Fortran, Erlang, respectively).
Having drawn a pseudo-config on a piece of paper, carefully analyze it and, if everything is in order, format it as a valid YAML file.
You will have to solve the dilemma “tabs or spaces?” . Not in a global sense, but only at the level of your organization, or at least a project. It doesn’t matter if you use post-processing by the sed script, setting up text editors on programmers' machines, or taking receipts on strict observance of the linter’s instructions under the threat of dismissal, but all members of your team who are in any way concerned with YAML must Use only spaces (as required by the YAML specification).
In any normal text editor, you can set AutoCorrect tabs to a specified number of spaces, so you can’t be afraid of the Tab riots.
As every hater of YAML is well aware, the difference between tabs and spaces is not visible on the screen. And when something is not visible, this is usually remembered last, after it has been sorted, checked and eliminated all other possible problems. The hour of time killed to search for a tab curve or a block of spaces simply cries out that you urgently need to create a policy for using one or the other, and then implement a reinforced concrete verification of its compliance (for example, through a Git hook for forced run through the linter).
Some people like to write in YAML because it emphasizes structure. At the same time, they actively use indents to highlight data blocks. This is a kind of scam to simulate markup languages ​​that use explicit delimiters.
Here is an example of such structuring from the Ansible documentation :
This option helps someone to decompose the YAML structure in their heads, while on the contrary, it annoys others with a lot of indentions that are unnecessary in their opinion.
But if you are the owner of a YAML document and are responsible for maintaining it, then you and only you must determine how to use the indentation. If you are annoyed by large indentation, keep them to the minimum possible according to the YAML specification. For example, the above file from the Ansible documentation can be rewritten without any loss like this:
If you constantly repeat the same mistakes when filling out the YAML file, it makes sense to insert a blank template into it as a comment. Then next time it will be possible to simply copy this workpiece and enter real data there, for example:
If the application does not hold you in a stranglehold, then it may be worth changing YAML to another format. Over time, the configuration files can outgrow themselves and then it is better to convert them into simple scripts in Lua or Python.
YAML is a great thing that many people love for its minimalism and simplicity, but this is far from the only tool in your arsenal. So sometimes you can refuse it. It’s easy for YAML to find parsing libraries, so if you offer convenient migration options, your users will survive this failure relatively painlessly.
If you can’t do without YAML, then take these 10 tips into service and defeat your dislike of YAML once and for all!
If YAML infuriates you, you can - and must! - Take the following 10 steps to reduce your irritation to an acceptable level and love YAML. As befits this list, our ten tips will be numbered from scratch, we add meditation and spiritual practices at will ;-)
0. Make your editor work
It doesn't matter what kind of text editor you have - for him, for sure, there is at least one plugin for working with YAML. If you don’t have one, find and install immediately. The time spent on searching and setting up will pay off many times each time you have to edit YAML.
For example, the Atom editor supports YAML by default, but GNU Emacs will have to install additional packages, for example, yaml-mode .
Emacs in YAML mode and display spaces.
If your favorite editor does not have YAML mode, then some of the problems can be solved by working with the settings. For example, the standard Gedit text editor for GNOME does not have YAML mode, but it defaults to highlighting YAML syntax and allows you to configure work with indentation:
Gedit indentation settings.
And the drawspaces plugin for Gedit displays spaces as dots, eliminating ambiguities with indentation levels.
In other words, take the time to study your favorite editor. Find out what he or his development community is offering to work with YAML, and use these features. You definitely will not regret it.
1. Use a linter (linter)
Ideally, programming and markup languages ​​use predictable syntax. Computers do a good job of predictability, so back in 1978 the concept of a linter appeared . If for 40 years of its existence it passed you and you still do not use the YAML-linter, then it's time to try yamllint.
You can install yamllint using the standard Linux package manager. For example, on Red Hat Enterprise Linux 8 or Fedora, this is done like this:
$ sudo dnf install yamllint
Then you just run yamllint, passing it the YAML file for verification. Here's what it looks like if you pass an error file to the linter:
$ yamllint errorprone.yaml errorprone.yaml 23:10 error syntax error: mapping values are not allowed here 23:11 error trailing spaces (trailing-spaces)
The numbers on the left are not time, but the coordinates of the error: row and column number. The error description may not tell you anything, but you know exactly where it is. Just look at this place in the code, and most likely, everything will become clear.
When yamllint does not find errors in the file, nothing is displayed. If you are afraid of such silence and want a little more feedback, then you can start the linter with the conditional echo command through the double ampersand (&&), like this:
$ yamllint perfect.yaml && echo "OK" OK
In POSIX, a double ampersand is triggered if and only if the previous command returns 0. And yamllint just returns the number of errors found, so this whole conditional construction works.
2. Write in Python, not YAML
If YAML really pisses you off, just don't write on it, literally. It happens that YAML is the only format that is perceived by the application. But in this case, it is not necessary to create a YAML file. Write on what you like, and then convert. For example, for Python there is an excellent pyyaml library and two whole conversion methods: self-conversion and conversion through scripts.
Self-conversion
In this case, the data file is also a Python script that generates YAML. This method is best suited for small data sets. You simply write JSON data to a Python variable, precede it with the import directive, and add three lines at the end of the file to implement the output.
#!/usr/bin/python3 import yaml d={ "glossary": { "title": "example glossary", "GlossDiv": { "title": "S", "GlossList": { "GlossEntry": { "ID": "SGML", "SortAs": "SGML", "GlossTerm": "Standard Generalized Markup Language", "Acronym": "SGML", "Abbrev": "ISO 8879:1986", "GlossDef": { "para": "A meta-markup language, used to create markup languages such as DocBook.", "GlossSeeAlso": ["GML", "XML"] }, "GlossSee": "markup" } } } } } f=open('output.yaml','w') f.write(yaml.dump(d)) f.close
Now run this file in Python and the output file is output.yaml:
$ python3 ./example.json $ cat output.yaml glossary: GlossDiv: GlossList: GlossEntry: Abbrev: ISO 8879:1986 Acronym: SGML GlossDef: GlossSeeAlso: [GML, XML] para: A meta-markup language, used to create markup languages such as DocBook. GlossSee: markup GlossTerm: Standard Generalized Markup Language ID: SGML SortAs: SGML title: S title: example glossary
This is absolutely correct YAML, but yamllint will warn you that it does not start with ---. Well, this can be easily fixed manually or slightly modified Python-script.
Convert via scripts
In this case, we first write in JSON and then run the converter in the form of a separate Python script, which yields YAML in the output. Compared to the previous one, this method scales better because the conversion is separated by data.
First, create the JSON file example.json, for example, you can take it on json.org :
{ "glossary": { "title": "example glossary", "GlossDiv": { "title": "S", "GlossList": { "GlossEntry": { "ID": "SGML", "SortAs": "SGML", "GlossTerm": "Standard Generalized Markup Language", "Acronym": "SGML", "Abbrev": "ISO 8879:1986", "GlossDef": { "para": "A meta-markup language, used to create markup languages such as DocBook.", "GlossSeeAlso": ["GML", "XML"] }, "GlossSee": "markup" } } } } }
Then create a simple script converter and save it under the name json2yaml.py. This script imports both modules - YAML and JSON Python, and also downloads the user specified JSON file, performs conversion and writes data to the output.yaml file.
#!/usr/bin/python3 import yaml import sys import json OUT=open('output.yaml','w') IN=open(sys.argv[1], 'r') JSON = json.load(IN) IN.close() yaml.dump(JSON, OUT) OUT.close()
Save this script in the system path and run as needed:
$ ~/bin/json2yaml.py example.json
3. Parse a lot and often
Sometimes it’s useful to look at the problem from a different angle. If you find it difficult to imagine the relationships between data in YAML, you can temporarily transform them into something more familiar.
For example, if you are comfortable working with dictionary lists or with JSON, then YAML can be converted to JSON with just two commands in the interactive Python shell. Suppose you have a mydata.yaml YAML file, then here is how it would look:
$ python3 >>> f=open('mydata.yaml','r') >>> yaml.load(f) {'document': 34843, 'date': datetime.date(2019, 5, 23), 'bill-to': {'given': 'Seth', 'family': 'Kenlon', 'address': {'street': '51b Mornington Road\n', 'city': 'Brooklyn', 'state': 'Wellington', 'postal': 6021, 'country': 'NZ'}}, 'words': 938, 'comments': 'Good article. Could be better.'}
A lot of other examples can be found on this subject. In addition, there are many online converters and local parsers. So do not hesitate to reformat the data when you see in them only an incomprehensible hash.
4. Read specs
Returning to YAML after a long break, it’s useful to go to yaml.org and re-read the specs. If you have difficulties with YAML, but your hands haven’t reached the specification, then it’s time to correct this situation. The specs are surprisingly easy to write, and the syntax requirements are illustrated by a large number of examples in Chapter 6 .
5. Pseudoconfigs
When writing a book or article, it is always useful to first draft a preliminary plan, at least in the form of a table of contents. So it is with YAML. Most likely, you can imagine what data you need to write to the YAML file, but you don’t really understand how to connect them to each other. Therefore, before sculpting YAML, draw a pseudo-config.
Pseudo-config is similar to pseudo-code, where you do not have to worry about structure or indentation, parent-child relationships, inheritance and nesting. So it is here: you draw iterations of data as they arise in your head.
A pseudo-config listing programmers (Martin and Tabitha) and their skills (programming languages: Python, Perl, Pascal and Lisp, Fortran, Erlang, respectively).
Having drawn a pseudo-config on a piece of paper, carefully analyze it and, if everything is in order, format it as a valid YAML file.
6. The tab or dilemma dilemma
You will have to solve the dilemma “tabs or spaces?” . Not in a global sense, but only at the level of your organization, or at least a project. It doesn’t matter if you use post-processing by the sed script, setting up text editors on programmers' machines, or taking receipts on strict observance of the linter’s instructions under the threat of dismissal, but all members of your team who are in any way concerned with YAML must Use only spaces (as required by the YAML specification).
In any normal text editor, you can set AutoCorrect tabs to a specified number of spaces, so you can’t be afraid of the Tab riots.
As every hater of YAML is well aware, the difference between tabs and spaces is not visible on the screen. And when something is not visible, this is usually remembered last, after it has been sorted, checked and eliminated all other possible problems. The hour of time killed to search for a tab curve or a block of spaces simply cries out that you urgently need to create a policy for using one or the other, and then implement a reinforced concrete verification of its compliance (for example, through a Git hook for forced run through the linter).
7. Better less is better (or more is less)
Some people like to write in YAML because it emphasizes structure. At the same time, they actively use indents to highlight data blocks. This is a kind of scam to simulate markup languages ​​that use explicit delimiters.
Here is an example of such structuring from the Ansible documentation :
# Employee records - martin: name: Martin D'vloper job: Developer skills: - python - perl - pascal - tabitha: name: Tabitha Bitumen job: Developer skills: - lisp - fortran - erlang
This option helps someone to decompose the YAML structure in their heads, while on the contrary, it annoys others with a lot of indentions that are unnecessary in their opinion.
But if you are the owner of a YAML document and are responsible for maintaining it, then you and only you must determine how to use the indentation. If you are annoyed by large indentation, keep them to the minimum possible according to the YAML specification. For example, the above file from the Ansible documentation can be rewritten without any loss like this:
--- - martin: name: Martin D'vloper job: Developer skills: - python - perl - pascal - tabitha: name: Tabitha Bitumen job: Developer skills: - lisp - fortran - erlang
8. Use blanks
If you constantly repeat the same mistakes when filling out the YAML file, it makes sense to insert a blank template into it as a comment. Then next time it will be possible to simply copy this workpiece and enter real data there, for example:
--- # - <common name>: # name: Given Surname # job: JOB # skills: # - LANG - martin: name: Martin D'vloper job: Developer skills: - python - perl - pascal - tabitha: name: Tabitha Bitumen job: Developer skills: - lisp - fortran - erlang
9. Use something else
If the application does not hold you in a stranglehold, then it may be worth changing YAML to another format. Over time, the configuration files can outgrow themselves and then it is better to convert them into simple scripts in Lua or Python.
YAML is a great thing that many people love for its minimalism and simplicity, but this is far from the only tool in your arsenal. So sometimes you can refuse it. It’s easy for YAML to find parsing libraries, so if you offer convenient migration options, your users will survive this failure relatively painlessly.
If you can’t do without YAML, then take these 10 tips into service and defeat your dislike of YAML once and for all!
All Articles