We're finally at the point where our infrastructure is cleaned up enough that getting usable signal through all the logspew is finally possible. After comparing a few alternatives, including Azure Sentinel and the revered Splunk, we decided on FortiSIEM. Setup of any monitoring platform from scratch is some of the most tedious IT work there is, but it also gives you a clear understanding of the how and the why of the product.
A few things that I've found incredibly frustrating through the first few hours of setup are still frustrating. After poking around and understanding how the parts are interconnected a bit more, some of the things I found frustrating now seem sensible. The site design could be a bit less Java app from 2003, but it's fine.
The Good
Installing the Windows agent
Installing the Windows Agent is a breeze. Create a local account on the SIEM with Agent Install rights. Copy the installer file along with the template XML that includes the collector's address, and the Agent Installer credentials to the Windows server you're installing on, and install with /passive and reboot.
Configuring Windows client groups is easy as well, if not a little repetitive.
The Bad
Updating Agent Configuration requires a reboot
Making changes to an agent's grouping requires that the server running the agent be rebooted. Either that or it takes over an hour to refresh, which I feel is kinda too long for a security product to default to.
The Ugly
Fortinet's documentation for FortiSIEM is terrible.
It's basically a step by step of "Fill out the field with the field," with zero insight into what it's actually asking for. When selecting Windows Event exclusions or inclusions, the documentation says "Enter the events to be included under Include Event and the ones to exclude under Exclude Event." It doesn't say Event source, Event type, Event ID, and the interface doesn't give any hints either. It's not until you enter something other than a comma separated list of event IDs that you get an error indicating what it expects.
While having to do this and move on isn't a big deal, it's emblematic of the design of the product, and the design of the documentation. There's sometimes a breakdown of what the field is supposed to be, but usually when it's unneeded; "DNS Server - Enter the DNS server IP address." It's just cumbersome enough that you can bump your way through the slightly ambiguous portions of the setup, but mostly it feels like I should probably just be paying their Professional Services to set this up. The search on the documentation site is likewise awful, frequently dropping me from 6.1.0 documentation into 5.2.x for no apparent reason, even when there was a 6.1 page available.
The feedback from the GUI is terrible
Error messages should always tell the user something that they can use to figure out what went wrong. "Test failed," or some other utterly useless message pops up very frequently leaving me with no idea if the test failed because my credentials were invalid, because the connection timed out, because it was refused, because a DNS lookup failed, because the SIEM couldn't bind to eth0, or who knows what. This leads to more bumping around trying to figure out what's wrong than I should have to do. Instead of going and checking to see if DNS had finished replicating, I instead go and redo the credential setup - Nope - go change the TLS settings - Nope - restart the client that's not checking in - Nope.
Likewise, I use a pseudo diceware password generator, with occasional numbers/symbols tossed in so that we don't have everyone walking around with Summer2020 as their password for the next 15 years. Being told that a password like: surf 02067 CHOMP ' alien } ocean cameo" can't be set because "Password must be between 8-64 characters, with at least 1 letter, 1 number and 1 special character (e.g. $*&%)" is very not helpful.
Spaces are valid characters. It's 2021, this is bad passwording Fortinet. Not only that, but there's zero indication of what caused the failure. User interfaces that obfuscate, or fail to show useful feedback are user-hostile.