Musings tagged as config

This is a followup to nginx RTMP Streaming With Simple Authentication.

Last time we covered a very basic setup with a hardcoded passkey. Multiple people have contacted me so far requesting an explanation on how to move towards a slightly more sophisticated authentication setup. Usually involving a php script to authenticate against. Maybe you want to use an existing mySQL or mariaDB database to set up users and channels? Fear not, this is not that complicated to start out with.

Server side configuration

Starting from the old example, we set up a basic rtmp section:

rtmp {
  server {
    listen 1935;
	ping 30s;
	notify_method get;
	  
	application stream {
	  live on;
	  on_publish http://yourdomain.com/rtmp_auth.php;
	  record off;
	}
  }
}

The on_publish command can point to any web address that you like. It could be supplied via the nginx server as well or you could use an apache2 instance for that. You can also use a completely different server if you wish. For now, we assume that there is a php script rtmp_auth.php which sits in the webroot of your webserver.

The above line will do the following:

  • as soon as someone tries to publish a stream to your domain, the nginx rtmp module will issue a HTTP POST request to the on_publish url.
  • nginx will supply the script with the get variable “name” and fill it with whatever comes directly after your initial stream url
  • it will also pass on any further GET style variables via the standard ?var1=value1&var2=value2 syntax.
  • it will wait for a HTTP return code which either tells it that everything is fine and streaming should commence (201) or that something went wront and it should drop the connection (404)

Suppose someone uses the following url to connect to your rtmp server:

rtmp://yourdomain.com/stream/john?psk=supersecret

nginx will then call your rtmp_auth.php script like this:

http://yourdomain.com/rtmp_auth.php?name=john&psk=supersecret

Inside of your php script you then have access to the $_POST array which holds your values and you can do whatever you want with them. In the following example we will use a php array $valid_users to hold a list of allowed users and passwords. Of course, you could instead connect to a database and query for the username and password. The interesting part is all in the if-statement which follows after that.

<?php
$username = $_POST["name"]; # in our current example, this will be 'john'
$password = $_POST["psk"]; # in our current example, this will be 'supersecret'

$valid_users = array("john" => "supersecret",
                     "winnie" => "thepooh",
					 "batman" => "nananananananana");

if ($valid_users[$username] == $password) {
  http_response_code(201); # return 201 "Created"
} else {
  http_response_code(404); # return 404 "Not Found"
}
?>

With this code, if the credentials check out, we return a 201 status code which tells nginx that whoever tries to connect is allowed to stream. If they do not, we issue a 404 and tell the client to get lost.

Client side configuration

Let us suppose your streaming client uses open broadcaster studio (OBS), which is a free and open source streaming utility which works with pretty much all major streaming sites and can also be configured to a custom site.

In OBS, you set the stream type to Custom Streaming Server and as the url you would use rtmp://yourdomain.com/stream/. As the stream key you would set john?psk=supersecret or any other username / password combination. If you want to supply more information you can supply more GET style variables via appending &var1=value1&var2=value2 and so on. Anything you write before the ? in the stream key field will end up in the variable $_POST["name"] inside your php script.

Sadly, SSL support for RTMP and also for the internal on_publishrequest is still somewhat lacking. So keep in mind that all of this stuff is plaintext authentication. You might not want to use a username / password combo directly but rather a streaming hash that you can allocate via a database and which has to be requested by the user beforehand. At least this way, if someone captures the data, they do not know the login credentials of your users.

You could for example generate a 64 character hash, put it into an SQL table and assign it to a user of your site. Then the user just enters this hash in OBS into the stream key field. Your php script will then be called and the variable $_POST["name"] will hold this hash. After that you can connect to your database, check out whether the hash exists or not and maybe even set up some notification on your website that user john is now streaming. Just be sure to finish with a 201 or 404 code in order to let nginx-rtmp know what it should do about the connection attempt.

In the end you can make things as complex as you want. You can use the same method for other nginx-rtmp directives such as: on_play, on_done, on_update and many more. Check them out at the nginx-rtmp wiki page.

Update: A kind reader has informed me that newer versions of the nginx rtmp plugin no longer use a GET but a POST request to call the URL you specify in the on_publish, on_play, etc directives. I have updated the code to reflect the changes.

I’ve been looking for a simple way to utilize RTMP streaming with OBS Studio without having to resort to bloated websites like twitch. RTMP streams you can just play using your media player of choice (usually), for example VLC. In addition this can be useful to collaborate with colleagues, as you can stream not just a single window but also your whole desktop. nginx has access to a RTMP module (surprisingly called nginx-rtmp-module ^^) which they say does not do authentication but just the streaming. However, it features certain event calls like on_publish or on_play. After digging into it for a bit there is actually a super easy way to do a very simple authentication scheme which could be extended by any script you like.

All you need is first the RTMP block in your nginx config file:

rtmp {
  server {
    listen 1935;
	ping 30s;
	notify_method get;
			  
	application stream {
	  live on;
      on_publish http://localhost[:port]/auth;
      on_play http://localhost[:port]/auth;
	  record off;
	}
  }
}

This just sets up the stream and then forces to trigger a certain url on_publish (i.e. if someone wants to stream to the server) and on_play (i.e. someone trying to play back the stream). Now, the stream or play request will only be accepted if the url given returns a HTTP 2xx status code, else the connection will be dropped.

We can work this to our advantage and just set up a quick and dirty server in the http section of nginx that directly checks for a given secret:

server {
  listen <port>;
  location /auth {
  if ($arg_psk = 'totallysecretpassword') {
    return 201;
  }
  return 404;
}

You could also implement multiple urls with different secrets for streaming and playing or you could let a script of your choosing answer the url request and check for a username and a password or id in a database. Once all is done you can access the stream via rtmp://host.tld/stream/user?psk=totallysecretpassword. Of course this sends your password via plaintext so you should take precautions and if your software supports it use rtmp via ssl/TLS.

UPDATE: there is now a followup article on how to realize a slightly more sophisticated authentication setup

In order to get the new site up and running and switching to Hugo a few things were missing.

Pagination

I did not want a Next and Previous navigation as this gets really annoying with lots of pages. I wanted a list of pages, not too long through, and the ability to quickly traverse pages.

First we define the parameters for number of shown pages and at which position the active page is supposed to be located. This neatly describes the number of previous and following pages to generate. This goes into the config.toml site configuration:

[params]
	pgtrspan = 6
	pgtractive = 3

Then this goes into a partial template which you can include to show the paginator, lets call this pagenav.html:

<div id="page-bar">
  {{ $curpage := .Paginator.PageNumber }}
  {{ $lastpage := .Paginator.TotalPages }}

  <!-- if we have previous pages show quick rewind buttons -->
  {{ if .Paginator.HasPrev }}
  <div class="pagenumber">
   <a href="{{ .Paginator.First.URL }}"><span class="oi" data-glyph="caret-left" title="previous page" aria-hidden="true"></span><span class="oi" data-glyph="caret-left" title="previous page" aria-hidden="true"></span></a>
  </div>
  <div class="pagenumber">
   <a href="{{ .Paginator.Prev.URL }}"><span class="oi" data-glyph="caret-left" title="previous page" aria-hidden="true"></span></a>
  </div>
  {{ end }}

  <!-- get the width of the navpage in number of pages -->
  {{ $pgtrspan := $.Site.Params.pgtrspan }}

  <!-- get the index of the active page -->
  {{ $pgtractive := $.Site.Params.pgtractive }}

  
  {{ $lowerbound := sub $curpage (sub $pgtrspan (add $pgtractive 1)) }}
  {{ $upperbound := add $curpage (sub $pgtrspan $pgtractive) }}

  {{ if le $upperbound $pgtrspan }}
    {{ $.Scratch.Set "lowerbound" 1 }}
    {{ $.Scratch.Set "upperbound" $pgtrspan }}
  {{ else if gt $upperbound $lastpage }}
    {{ $.Scratch.Set "lowerbound" (add (sub $lastpage $pgtrspan) 1) }}
    {{ $.Scratch.Set "upperbound" $lastpage }}
  {{ else }}
    {{ $.Scratch.Set "lowerbound" $lowerbound }}
    {{ $.Scratch.Set "upperbound" $upperbound }}
  {{ end }}

  <!-- loop through paginator pages and only display stuff within the boundaries -->
  {{ range $id, $pager := .Paginator.Pagers }}
  {{ if and (ge $pager.PageNumber ($.Scratch.Get "lowerbound")) (le $pager.PageNumber ($.Scratch.Get "upperbound")) }}
  <div class="pagenumber">
    {{ if eq $curpage $pager.PageNumber }}
      {{ $pager.PageNumber }}
    {{ else }}
      <a href="{{ $pager.URL }}">{{ $pager.PageNumber }}</a>
    {{ end }}
  </div>
  {{ end }}
  {{ end }}

  <!-- if we have following pages show quick forward buttons -->
  {{ if .Paginator.HasNext }}
  <div class="pagenumber">
   <a href="{{ .Paginator.Next.URL }}"><span class="oi" data-glyph="caret-right" title="next page" aria-hidden="true"></span></a>
  </div>
  <div class="pagenumber">
   <a href="{{ .Paginator.Last.URL }}"><span class="oi" data-glyph="caret-right" title="previous page" aria-hidden="true"></span><span class="oi" data-glyph="caret-right" title="previous page" aria-hidden="true"></span></a>
  </div>
  {{ end }}
</div>

So, finally just put this below whatever paginator range you like:

{{ $paginator := .Paginate .Data.Pages }}
{{ range $paginator.Pages }}
<article>
    <h2><a href="{{ .Permalink }}">{{ .Title }}</a></h2>
    {{ .Content }}
</article>
{{ end }}
{{ partial "pagenav.html" . }}

And presto! Proper pagination bar.

Post Archive

Now the archive business is rather simple but took me a while to figure out as the documentation is really lacking in this regard. There are a couple of examples you can find online but none of them really did it the exact way I wanted it. Yearly archives are fairly easy to do but scrolling through a year worth of posts can be just as annoying if the post volume is high enough. So, ideally I want monthly archives but grouped by year. Leveraging this with Hugo’s taxonomy system is a tiny bit annoying but once it works it is perfectly serviceable.

First of all, add an archive taxonomy to your site’s config.toml

[taxonomies]
	archive = "archive"

Next, on all your posts, add the year and month as seperate taxonomy terms in the front matter

+++
title = "Wabbit"
date = 2017-12-09T16:59:48+01:00
archive = ["2017","2017-12"]
+++

So, the problem is how do we separate the YYYY terms from the YYYY-MM terms? Simple answer is, we don’t. When do we want the archive to start? That’s right, from the year and month we made the first post. When do we want it to end? How about at the latest post? So we create the termlisting taxonomy file as archive.terms.html which goes into one of the proper folders, in my case I use the layouts/taxonomy folder in the theme.

We then loop over all the years between the oldest and the newest post and after printing the list item for said year we loop through the terms. Only if a term matches the YYYY-MM format with YYYY matching the year we are currently indexing will it be printed. And since printing the term as is would be redundant as it contains the year again, we beautify it by use of dateFormat.

By the way, if someone knows a more elegant way in Hugo/Go-Template syntax to return true or false depending on whether a RegEx matches a string please drop me a line. The combination of len, findRE and a comparison operation was the shortest way I could think of with the limited toolset you get.

{{ partial "header.html" . }}

<section id="archive">
  <h1>Archive</h1>
  <ul>
    {{ $data := .Data }}
    {{ $firstyear := (index .Data.Pages.ByDate 1).Date.Format "2006" }}
    {{ $currentyear := now.Format "2006" }}

    {{ range $i,$year := (seq $currentyear $firstyear) }}
      <li><a href="/{{ $data.Plural }}/{{ $year }}">{{ $year }}</a><ul>
      {{ range $key, $value := $data.Terms.Alphabetical.Reverse }}
        {{ if gt (len (findRE (print "^" $year "-\\d\\d$") $value.Name)) 0 }}
          <li><a href="/{{ $data.Plural }}/{{ $value.Name }}">{{ dateFormat "January" (print $value.Name "-01") }}</a> ({{ $value.Count }})</li>
	{{ end }}
      {{ end }}
      </ul></li>
    {{ end }}
  </ul>
</section>

{{ partial "footer.html" . }}

Now, all that is left is to create archive.html which will be used to actually display the paginated list of articles for a year or for a month, depending on which gets selected by the viewer. We only check whether the term is in the YYYY or YYYY-MM format and output a header which is formatted accordingly. The rest is just simple iterating through the posts.

{{ partial "header.html" . }}

{{ $paginator := .Paginate (where .Data.Pages "Type" "musings") 10 }}
{{ $data := .Data }}

{{ if gt (len (findRE "\\d\\d\\d\\d-\\d\\d$" .Data.Term)) 0 }}
{{ $.Scratch.Set "heading" (print (dateFormat "January" (print $data.Term "-01")) " " (replaceRE "-\\d\\d$" "" $data.Term)) }}
{{ else }}
{{ $.Scratch.Set "heading" $data.Term }}
{{ end }}

<h1>Archive for {{ $.Scratch.Get "heading" }}</h1>
{{ range $paginator.Pages }}
<article>
  <h2><a href="{{ .Permalink }}">{{ .Title }}</a></h2>
    {{ .Content }}
</article>
{{ end }}
{{ partial "pagenav.html" . }}

{{ partial "footer.html" . }}

And we’re done.

1