mardi 12 mars 2013

pgBadger 3 : How to use the parallel mode

Just like I said in a previous blog post : the pgBadger projet is moving fast ! Version 3 was released a few days ago with a major improvements : With the new parallel mode, you can now launch mutiple pgBagder process at once !

Beware !

In others word, pgBadger is not bound to one cpu anymore. If you have multiple cores, you can now uses them to give more power and decrease the processing time. All you have to do is add the "-j N" option in the command line, where N is the number of cores you want to use. For instance :

$ pgbadger -j 4 /var/log/postgresql/postgresql-2013-02-*

Quick benchmark : We had 5 log files with a total volume of 9.5 GB. Here's the results

  • -j 1 : 1h41m18s
  • -j 2 : 50m25s
  • -j 4 : 25m39s
  • -j 8 : 15m58s


However it's important to note that the parallel mode has a little drawback. With this method, some queries may be truncated. If you enable N cores, then result may differ in a maximum of N queries per log file.

However, this is a minor issue: parallel mode is interesting if you have millions of queries to analyze. And if you have millions of queries in a log file, you can afford to loose a few as it's quite unlikely that the lost queries would have changed the overall results.

However, to avoid this problem, you can use the pgBadger "per-file parallel mode" to analyze your logs but with lower performance than the standard parallel mode. To enable this behaviour, you have to use the "-J N" option instead of "-j N". In per-file mode, the performances start being really interesting when there's hundreds of small log files (e.g. 10MB rotation size limit) and with at least 8 cores.

We believe this new improvement will be a significant change for pgBadger users. If you have used the parallel mode, please send us feed back on our mailing list or via the pgBadger twitter account !

And by the way, this is not the only big feature in pgBadger v3 ! For the complete list of changes, please checkout the release note on github

And of course stay tuned.... because version 4 is coming fast :-)

PG Day France 2013 : Encore quelques jours pour soumettre vos propositions

Cette année encore la communauté francophone de PostgreSQL se réunira pour le PG Day France qui se tiendra à Nantes le 13 juin 2013. L'occasion parfaite pour présenter une étude de cas, un projet en cours de développement ou une fonctionnalité de PostgreSQL ! L'appel à orateurs se terminera dans  […]

Lire la suite

jeudi 3 janvier 2013

Refonte de pgBadger en HTML5 : premier aperçu


PgBadger est un outil puissant pour optimiser vos serveurs PostgreSQL. À partir des fichiers log de vos serveurs, pgBadger va analyser le trafic SQL et produire une profusion de statistiques (les plus longues requêtes, le ratio lecture/écriture, etc.) Tout ceci est très utile, mais la quantité de  […]

Lire la suite

pgBadger HTML5 overhaul : preview


pgBadger is a powerful tool to optimize your PostgreSQL servers. Using your server log files, pgBadger will analyze the SQL traffic and produce a profusion of statistics (longest queries, read/write ratio, etc.). This is very helpful but the amount of data generated is so big that the pgbadger  […]

Lire la suite

mercredi 14 novembre 2012

pgBadger is moving fast

When the pgBadger project started 6 month ago, our primary goal was to build a better performance tool for PostgreSQL. We also wanted it to be an open and dynamic project, built with the help of the Postgres community. So far the result is beyond our expectations : we've received dozens of user  […]

Lire la suite

mardi 13 novembre 2012

FOSDEM PGDay et Devroom 2013 - Annonce et Appel à Orateurs

Les personnes présentes à Prague pour 2012 ont déjà eu la primeur de cette annonce. Voici quelques détails complémentaires. Le PG Day du FOSDEM est une journée de conférence qui se tiendra juste avant le FOSDEM à Bruxelles le 1er février 2013. Il s'agit d'un événement dédié à PostgreSQL  […]

Lire la suite

vendredi 12 octobre 2012

PostgreSQL Session #4 : From Oracle to PostgreSQL

As a nice coincidence the 4th PostgreSQL Session held Thursday, October 4 which was also the last day of Oracle Open World conference. Suffice to say that the two events didn't share the same definition of the word "Open"! As for the previous sessions organized by Dalibo, the room was full  […]

Lire la suite

mercredi 10 octobre 2012

Session PostgreSQL #4 : Migrer d'Oracle à PostgreSQL

Hasard du calendrier la 4eme Session PostgreSQL se tenait jeudi 4 octobre, dernier jour de la conférence Oracle Open World. Autant dire tout de suite que les deux événements n'avaient pas la même définition du mot "Open" ! Comme à chaque session, la salle était pleine pour écouter les  […]

Lire la suite

- page 1 de 7