Max Kostikov - kostikov@tiksi.net

Max Kostikov

kostikov@tiksi.net

IT voodoo

Location:: Russia

Gender:: Мужчина

Homepage:: https://kostikov.co

Max Kostikov

Königsberg, Russia, Sat, 26 Jan 2019 23:10:51 +0200 last edited: Sat, 26 Jan 2019 23:11:18 +0200

!Hubzilla Development @Mark Nowiasz
I think we need to discuss this proposal (and related merge request).
https://framagit.org/hubzilla/core/issues/1330
I think this can lower overall system security and at the same time will give around nothing for performance.
Any thoughts?

show all 25 comments

phani (Hubzilla)

Sun, 27 Jan 2019 09:20:54 +0200

the only thing i've seen that looks insecure so far is having sessions stored in a world-writable place like /tmp. that could easily be avoided by creating a sub-dir under /tmp that's not completely open.

re. performance, i can't really test this myself since i'm on a one-man-hub that runs on SSD, not rotating metal, and any performance gain would be theoretical only.

so this isn't a 'real world problem' for me, but i find it interesting nonetheless. are there any other dangers when storing sessions not in the DB? and those bigger hubs that are there, what hardware do they run on? would it be a noticeable performance gain or not to write sessions to file instead of DB?

Max Kostikov

Sun, 27 Jan 2019 10:20:26 +0200

This is my point in brief.
1. Performance. Both session storage in file and database storage involve disk operations, and both have in-memory caching mechanisms so this shouldn't have any difference.
2. Reliability. File based sessions will be lost on every PHP application server restart and this disadvantage is absent when sessions stored in a database.
3. Security. Sessions are superglobal so it potentialy opens a ways for hijacking from any other PHP process even if it unleated to Hubzilla. Database storage gives additional security level thru SQL server authentication mechanisms.

phani (Hubzilla)

Sun, 27 Jan 2019 10:32:57 +0200

thank you; makes sense to me and might have to be revisited if & when huge hubs have to deal with performance problems. personally i like small hubs better anyway since that's more decentralized than having big hubs with hundreds or thousands of users.

Mario Vavti

Sun, 27 Jan 2019 10:59:44 +0200

I have tested the patch and have not seen an improvement in performance on my system.

If there is a situation where there is an advantage in file based session storage and the mentioned security loss is acceptable (single user setups?) i would suggest to make it pluggable via an addon.

M. Dent

Sun, 27 Jan 2019 13:13:00 +0200

Generally I agree with the add-on approach, my philosophy is, I'd rather see a new hook than additional logic in core.

As to performance, I agree that moving to file based sessions seems to be a wash - but it's really going to depend on hardware setup. If the files are on SSD and your db on spinning disks or even on an external server which may have latency issues, the boost could be huge

As to security, yes, the default here is in some manner less secure - and given that it is likely that HZ will be installed in a shared hosting environment (not even VPS), which is a security nightmare in itself , we can't assume competence or in some cases even ability to create/change directory permissions.

Clearly this would really only be useful for very busy hubs but really it merely implements existing PHP functionality to specify alternative handlers.

For those reasons I really like the concept and am very thankful that Mark is thinking about performance. A hook + add-on approach, however, would be preferable in my book.

poVoq

Mon, 28 Jan 2019 10:12:05 +0200

I don't see why this could not be added as a non-default option. At least for our setup it would probably result is a speed boost (SSD, but database on a seperate server, might even be possible to use a tempfs folder).

Mark Nowiasz

Mon, 28 Jan 2019 14:30:40 +0200

I think you guys overlook one import fact: /tmp now usually is tmpfs, i.e. a a RAM disk, which is far, far superior in speed than even a SSD. Of course, if you'd use a "real" file system for session storage that would amount to nothing - it might even be slower than using a database.

However, as I stated in the article leading to the patch: If you haven't got a tmpfs, you also could use memcached or Redis as a session handler - and believe me, this is the way to go for every efficient site I've ever adminstrated.

I've stated all this in my article - tmpfs, memcached, Redis:

Mark Nowiasz schrieb den folgenden Artikel Fri, 25 Jan 2019 12:05:10 +0100

Activating Warp Speed for Hubzilla's Database

How to significantly gain speed and perfomance for Hubzilla by just some minor adjustments.
Note: The DB section is about MySQL/MariaDB. However, PostgresQL will be similar effected. The solultion is database agnostic.

Artikel ansehen

Zusammenfassung ansehen

Preface

As my hub (hub.netzgemeinde.eu) grew I noticed sizable performance issues (including timeouts to hubzilla), originating from the database. The MariaDB in question was on a conventional SAS RAID, therefore the logical step was moving the database to a SSD host, making the old host a slave to the new db master. This significantly improved speed.

However, I noticed something odd: The slave (the former master) was experiencing rather high iowait time (up to 20%). Digging deeper into the issue I found that php's sessions were being stored in the database. And that is a very bad idea, it's one of the surest ways (apart from missing indexes and so) to bring the database to it's knees!

Let me tell you a story: There was this sports club which had an online shop for it's merchandise which also stored it's sessions in a database. Now against all odds the club won the cup, and guess what happened and who was on on-call duty that evening? I tried to mitigate this by adding 64 (yes, 64) cores to the database server, it just gobbled them up in microseconds so I told the customer that there was nothing we could do about this, just wait until the rush was over - and then make some optimizations, including removing the sessions from the database. (This had to be done by developers). After that (and a couple of other optimizations) the shop went fine. This was the worst offender I've seen, yet by far not the only one.

Back to Hubzilla

The slave

I've changed the replication to ignore the session table:

replicate-ignore-table = hub_netzgemeinde_eu.session

Usually this is something that makes every experienced DBA (including me) cringe - partial replication is one sure way to break your slave/backups, but in this case it's OK, since there is no point in having the session table replicated or backupped - If you have to perform disaster recovery like promoting the slave to master or restoring a backup, the sessions are the least of your concerns

Let's have a look at the perfomance data - io wait:

Guess the time whan I disabled session's replication?

This is rather compelling, isn't it?
This fixed the iowait issue on the slave's host. Now to the master.

The master

Now it's time to fix this issue once and for all: Take the session handling out of the database

There are basically two options: Use file based session handling or external session storage. Both have advantage and disadvantages:

File based session handling Store your sessions in files residing /tmp. Some time ago this was considered to be a bad/slow solution (after all, a database is faster, eh?) However, tmp now usually has got tmpfs as a filesystem, which means /tmp is a RAM disk, which is much, much faster than a database. (If your tmp is not having a tmpfs, try the other option). There are two caveats, though: When the machine where your PHP interpreter (either fpm or mod_php) runs on has to reboot (either due to planned maintenance or powerloss) the sessions will be gone, the users will have to log in again That's an acceptable price to pay. The second one concerns scaling issue: If you run a high-performance setup (i.e. load balancer, more than one webserver) this can't be used since all application servers have to access the shared sessions.
External session storage You run a small key/value in-memory database on another host. The usual ones are memcache or Redis (I myself haven't got mixed experiences with Redis as as session store, but your mileage may vary). All application servers (Apache with PHP) can access the storage, so this is the way to go if you have more than one application server. If you use Redis, the session can also be saved in regular intervals on the disk, therefore you won't lose the sessions if your session storage goes down (reboot, update, whatver), however you will lose some speed advantage. I myself would recommend memcache, it's blazingly fast and stores the session ony in memory, but feel free to toy with Redis.

Setting it up in Hubzilla

Unfortunately Hubzilla doesn't support changing the session handler out of the box. I've written a small patch (which I will submit for review) to enable ist (it's attached to this article):

cd <your instance>/Zotlabs/Web
(copy Session.patch here)
patch < Session.patch

After that you can switch the session handler, by default it's still database, so the patch won't do any harm. If you want to change the handler change .htconfig.php in Hubzilla's base directory:

App::$config["system"]['session_custom'] = true;
App::$config["system"]['session_save_handler'] = "files";
App::$config["system"]['session_save_path'] = "/tmp"

(Or whatever you prefer - memcached, redis,..)

After that all your former sessions will be unavailabe, i.e. users have to login again (including nomad's users), but after that things run rather smoothly.

About the performance gain

In my case it's hard to say, because the ssd is so fast - still the graph seems to indicate that even on a fast ssd (4 cores, 16 GB Ram, 10 GB Innodb buffer pool) there is a perfomance gain. I'd love to see if someone tries this on a slow machine - how much perfomance did you gain? I'd really be interested in feedback.

Conclusion

By changing Hubzilla's session handling one can gain a significant perfomance boost - the only disadvantage is that the sessions may be a little bit volatile (i.e. not reboot safe), but that's a small price to pay.

Edit 2019-01-26

I've expanded the patch making it possible to change session.gc_probability and session.gc_divisor (defaulting to 1 and 100 for custom session handlers):


App::$config["system"]['session_gc_probability'] = 1;
App::$config["system"]['session_gc_divisor'] = 100;

(Session2.patch)

https://hub.netzgemeinde.eu/articles/buckaroo/ef2110301dc5c6e5

About tmp's security: Systemd's PrivatTmp-Feature mitigates some security concerns:


drwx------ 3 root   root     60 23. Jan 09:22 systemd-private-09c107e582b24d5392f0473a37d902ae-apache2.service-mLRUr4
drwx------ 3 root   root     60 22. Jan 10:46 systemd-private-09c107e582b24d5392f0473a37d902ae-dovecot.service-WOAPei
drwx------ 3 root   root     60 22. Jan 08:50 systemd-private-09c107e582b24d5392f0473a37d902ae-openvpn-server@openvpn.midworld.de.service-8Zzruv
drwx------ 3 root   root     60 26. Jan 23:07 systemd-private-09c107e582b24d5392f0473a37d902ae-php-fpm@7.2.service-UL353C
drwx------ 3 root   root     60 22. Jan 10:46 systemd-private-09c107e582b24d5392f0473a37d902ae-postfix.service-SCu2Mb
drwx------ 3 root   root     60 22. Jan 08:50 systemd-private-09c107e5

But I really can't see how this can be implemented as a addon? I mean it's integrated into Hubzilla's core, it's non-invasive (if you don't add the new settings, nothing will be changed), and it's a very small change.

Mark Nowiasz

Mon, 28 Jan 2019 14:41:42 +0200

@Max Kostikov Unfortunately, this is not true:

1. Performance. Both session storage in file and database storage involve disk operations, and both have in-memory caching mechanisms so this shouldn't have any difference.

Caching is all well and good for read opertion - write operations (like UPDATE, INSERT, DELETE) have to be writte immediately to the disk, regardless of caching (innodb_buffer_pool, whaterver), otherwise the DB is not ACID compliant and there's risk of data loss. So each and every session update will(!) create a synced write on the disk, which is the reason that sessions stored in the database will bring the DB down - sooner or later, regarding on your setup, of course. Small hubs won't be affected, the more users you have the more it's going to become a problem. (And don't forget binlogs/WAL)

As for persistant sessions - in this case, if it's really an issue, you use memcached or Redis (either on a different machine or on the same one, Redis is able to write the data regularly - but not at once, like a ACID compliant DB - to the disk.

Max Kostikov

Mon, 28 Jan 2019 15:20:56 +0200

I think you guys overlook one import fact: /tmp now usually is tmpfs

There is a life outside Linux.

So each and every session update will(!) create a synced write on the disk

This is true. But how often it happens in Hubzilla? Have you any stats?
I suspect that session save should occur relatively rarely and in contrast with very often read.

But in common I agree that this can be useful for some installations especially where database located on remote server.

Mark Nowiasz

Mon, 28 Jan 2019 15:29:56 +0200

@Max Kostikov Not only that - I've got a zabbix graph which is included in my article. Look at the graph, look what happened (io wait wise) after I disabled the replication of the session table. There's a lot of session writing going on, You also could have a look at the binlogs/WAL (depending on your database) how much INSERT/UPDATE/DELETE statements there are.

So, let me see: I've got graphs, I've got speed gains and I've seen this behaviour lots of times in the past, causing shops to go down - why exaclty is it a problem giving an experienced user the choice of having his own session_handler, if he has got the need for it?

Mario Vavti

Mon, 28 Jan 2019 15:32:29 +0200

@Mark Nowiasz the implementation via addon would be very straight forward i think...

Mark Nowiasz

Mon, 28 Jan 2019 15:38:18 +0200

@Mario Vavti Be my guest

But I really don't think that it makes much sense changing these settings via web - I mean you don't setup the database via the web frontend, I think it makes more sense on this level - besides, if you do something wrong (like selecting a session handler which isn't supported - say, memcached without the pecl-memcached-module) you'd likely get a 500 HTTP error and you couln't reset the setting (since the web frontend would be unavailable)

Mario Vavti

Mon, 28 Jan 2019 15:51:14 +0200

@Mark Nowiasz it's not about where the settings are, its about where the code is.
You will be still able to set the settings in .htconfig.php

Mark Nowiasz

Mon, 28 Jan 2019 15:55:43 +0200

@Mario Vavti Ah, fair enough. Well, actually I don't care if this functionality is provided by my merge request or an addon - as long as I can setup my own session handler

Furthermore I'm not vain, so I don't care who claims the credit for the addon

Mario Vavti

Mon, 28 Jan 2019 16:05:04 +0200

Sigh... So you want me to do it?

It would be a nice opportunity for you to get to know the hubzilla addons system though
#justsayin'

Mark Nowiasz

Mon, 28 Jan 2019 16:12:54 +0200

My PHP is a little bit rusty, although I'm DevOps I'm more on the Ops side currently (hence my experience in tuning those things, especially online shops which do TV ads at prime time, so please believe me I know what I'm talking about

) I'd have to familiarize myself with PHP programming again (did this on and off since 1999), so I'd think you'd be more qualified than me

M. Dent

Mon, 28 Jan 2019 16:16:12 +0200

I can get it Mario. I already looked at doing it and I have a little extra time today.

Mario Vavti

Mon, 28 Jan 2019 16:32:18 +0200

Thanks @M. Dent !!!

Max Kostikov

Mon, 28 Jan 2019 17:18:20 +0200

BTW. I performed the audit on MySQL write activity into Hubzilla session table on @Zotum hub and got quite interesting results.
Here is a stats for about 40 mins. of monitoring

root@localhost [hubzilla]> SELECT * FROM `audit`;

+---------+------+------+------+
| tbl     | upd  | ins  | del  |
+---------+------+------+------+
| session | 2014 | 1144 | 1054 |
+---------+------+------+------+
1 row in set (0.00 sec)

Amazing.

Mark Nowiasz

Mon, 28 Jan 2019 17:39:57 +0200

@Max Kostikov This would amount to about 2 synced(!) writes/second - which is significant, especially if you haven't got a fast ssd. It's even worse in my case if you've got replication - because there's only one thread per database (even if you've got parallel replication enabled) so multiple cores won't help at all.

Max Kostikov

Mon, 28 Jan 2019 17:47:36 +0200 last edited: Mon, 28 Jan 2019 17:55:40 +0200

@Mark Nowiasz fsync will be performed on transaction close so this shouldn't be a problem but for replication / remote SQL server it can be so. Also fsync doesn't mean immediate disk write - in fact depend on filesystem it can happen even after a minute.
Anyway thank you for pointing this. Perhaps we need to investigate these results more deeply.

Mark Nowiasz

Mon, 28 Jan 2019 18:10:37 +0200

No problem

Oh, the next step in scaling hubzilla up/optimization would be read/write split: i.e. having one DB master (for writing) and one (or several for that matter) slaves for reading (this is another thing you do when you have a heavy workload). But I don't thinkt that it would come to that in the near future

Plus this would probably mean some serious code rewrites.

Mario Vavti

Tue, 29 Jan 2019 22:50:19 +0200

It turns out the code is not plugable because session init is called before load_hooks().

I suggest to implement this patch into core as suggested by @Mark Nowiasz in this case.

@Mark Nowiasz please open a MR to the dev branch or otherwise i can do that...

Mark Nowiasz

Tue, 29 Jan 2019 22:56:03 +0200

Done

Mario Vavti

Tue, 29 Jan 2019 22:59:43 +0200

Thanks!